Whenever I compile, i get this:
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 403 for URL: the link at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown
Source) at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown
Source) at java.net.URL.openStream(Unknown Source) at
readdata.aaa.main(aaa.java:15)
My script is:
package readdata;
import java.net.*;
import java.io.*;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
public class aaa
{
public static void main(String[] args) throws Exception {
URL oracle = new URL(" the link ");
BufferedReader in = new BufferedReader(
new InputStreamReader(oracle.openStream()));
String inputLine;
StringBuilder a = new StringBuilder();
while ((inputLine = in.readLine()) != null)
a.append(inputLine);
in.close();
int i = 0;
Pattern p = Pattern.compile("Open");
Matcher m = p.matcher( a );
while (m.find()) {
i++;
System.out.println(i);
}
}
}
Is there anyway I can bypass the cloudflare in order to read the data from the URL ?
Before
URL oracle = new URL(" the link ");
insert :
System.setProperty("http.agent", "Chrome");
That's probably because CloudFlare prevent from unknown agent requests so this code set the User-Agent to Chrome who is recognized by CloudFlare.
Related
I want to register and execute stored proc. I am using spring+Java with cosmos DB. Everytime I stop my application and restart it , it tried to create new sproc and since it already exists in cosmos DB it fails with below error . Is their any option available like "only create if not exist". I am fetching js file from src/main/resources folder.
I am following below doc to register the stored proc
https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-use-stored-procedures-triggers-udfs?tabs=java-sdk
#Configuration
public class StoredProcConfig
{
#Autowired
#Qualifier(BeansConstants.PAYMENT_CONTAINER)
CosmosContainer container;
#Bean
public CosmosStoredProcedureResponse registerSp() throws IOException
{
InputStream is = getFileFromResourceAsStream("storedProcedures/createStudent.js");
CosmosStoredProcedureProperties definition = new CosmosStoredProcedureProperties("spCreateToDoItems",
IOUtils.toString(is, StandardCharsets.UTF_8));
return container.getScripts().createStoredProcedure(definition);
}
private InputStream getFileFromResourceAsStream(String fileName)
{
// The class loader that loaded the class
ClassLoader classLoader = getClass().getClassLoader();
InputStream inputStream = classLoader.getResourceAsStream(fileName);
// the stream holding the file content
if (inputStream == null)
{
throw new IllegalArgumentException("file not found! " + fileName);
} else
{
return inputStream;
}
}
}
Error
Caused by: com.azure.cosmos.CosmosException: {"innerErrorMessage":"Message: {\"Errors\":[\"Resource with specified id, name, or unique index already exists.\"]}
Modify your registerSp() bean as below:
private static final Logger logger = LoggerFactory.getLogger(CosmosConfiguration.class);
#Bean
public CosmosStoredProcedureResponse registerSp() throws IOException
{
InputStream is = getFileFromResourceAsStream("storedProcedures/createStudent.js");
CosmosStoredProcedureProperties definition = new CosmosStoredProcedureProperties("spCreateToDoItems",
IOUtils.toString(is, StandardCharsets.UTF_8));
return createStoredProcedureIfNotExists(definition);
}
public CosmosStoredProcedureResponse createStoredProcedureIfNotExists(CosmosStoredProcedureProperties definition){
try {
CosmosStoredProcedureResponse storedProc = container.getScripts().getStoredProcedure(definition.getId()).read();
logger.info("found stored proc");
return storedProc;
}
catch (CosmosException e){
logger.info("stored proc not found, creating....");
return container.getScripts().createStoredProcedure(definition);
}
}
I have a program that connects to a user defined URL from a TextField and scrapes the images on that web page. The user defined URL is gotten from the textfield via .getText() and assigned to a String. The String is then used to connect to the Web page with JSoup and puts the webpage into a document.
String address = labelforAddress.getText();
try {
document = Jsoup.connect(address).get();
}catch(IOException ex){
ex.printStackTrace();
}
I've tried differently formatted URLS: "https://www.", "www.", "https://" but everything I use throws the malformed URL error.
Someone please show me how to get the text from the TextField the correct way.
Below is the entire code.
package sample;
import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.control.Label;
import javafx.scene.control.TextField;
import javafx.scene.layout.GridPane;
import javafx.stage.FileChooser;
import javafx.stage.Stage;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.*;
import java.net.URL;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class Main extends Application {
Document document;
LinkedList<String> imageURLList = new LinkedList<String>();
ArrayList<File> fileList = new ArrayList<File>();
int fileCount = 1;
#Override
public void start(Stage primaryStage) throws Exception{
Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
primaryStage.setTitle("Webpage Photo Scraper");
GridPane gp = new GridPane();
Label labelforAddress = new Label("URL");
GridPane.setConstraints(labelforAddress, 0,0);
TextField URLAddress = new TextField();
GridPane.setConstraints(URLAddress, 1,0);
Button scrape = new Button("Scrape for Photos");
GridPane.setConstraints(scrape, 0,1);
scrape.setOnAction(event->{
String address = labelforAddress.getText();
try {
document = Jsoup.connect(address).get();
}catch(IOException ex){
ex.printStackTrace();
}
Elements imgTags = document.getElementsByAttributeValueContaining("src", "/CharacterImages");
for(Element imgTag: imgTags){
imageURLList.add(imgTag.absUrl("src"));
}
for(String url: imageURLList){
File file = new File("C:\\Users\\Andrei\\Documents\\file" + fileCount + ".txt");
downloadFromURL(url, file);
fileList.add(file);
fileCount++;
}
});
Button exportToZipFile = new Button("Export to Zip File");
GridPane.setConstraints(exportToZipFile, 0,2);
exportToZipFile.setOnAction(event -> {
FileChooser fileChooser = new FileChooser();
FileChooser.ExtensionFilter exfilt = new FileChooser.ExtensionFilter("Zip Files", ".zip");
fileChooser.getExtensionFilters().add(exfilt);
try{
FileOutputStream fos = new FileOutputStream(fileChooser.showSaveDialog(primaryStage));
ZipOutputStream zipOut = new ZipOutputStream(fos);
for(int count = 0; count<=fileList.size()-1; count++){
File fileToZip = new File(String.valueOf(fileList.get(count)));
FileInputStream fis = new FileInputStream(fileToZip);
ZipEntry zipEntry = new ZipEntry(fileToZip.getName());
zipOut.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while((length = fis.read(bytes)) >= 0) {
zipOut.write(bytes, 0, length);
}
fis.close();
}
zipOut.close();
fos.close();
}catch(IOException e1){
e1.printStackTrace();
}
});
primaryStage.setScene(new Scene(gp, 300, 275));
primaryStage.show();
gp.getChildren().addAll(exportToZipFile, labelforAddress, scrape, URLAddress);
}
public static void downloadFromURL(String url, File file){
try {
URL Url = new URL(url);
BufferedInputStream bis = new BufferedInputStream(Url.openStream());
FileOutputStream fis = new FileOutputStream(file);
byte[] buffer = new byte[1024];
int count = 0;
while((count = bis.read(buffer, 0,1024)) !=-1){
fis.write(buffer, 0, count);
}
fis.close();
bis.close();
}catch(IOException e){
e.printStackTrace();
}
}
public static void main(String[] args) {
launch(args);
}
}
Your text field containing the value entered by user is stored in URLAddress object but you always try to get the url from labelforAddress object which is a label always containing "URL" text.
So the solution is to use:
String address = URLAddress.getText();
If you read carefully error message it would help you to find the cause, because it always displays the value it considers wrong. In this case I see:
Caused by: java.net.MalformedURLException: no protocol: URL
and it shows the unrecognized address is: URL.
If you encounter this kind of error next time try:
debugging the aplication in runtime to see values of each variable
logging variable values in the console to see if variables contain values you expect
I am trying to connect to a Docker UNIX domain socket using Netty. Here's my attempt so far.
#PostConstruct
public void init() throws Exception {
io.netty.bootstrap.Bootstrap bootstrap = new io.netty.bootstrap.Bootstrap();
bootstrap
.group(new NioEventLoopGroup())
.channel(NioSocketChannel.class)
.option(ChannelOption.SO_KEEPALIVE, true)
.remoteAddress(new DomainSocketAddress("/var/run/docker.sock"))
.handler(new ChannelInitializer<SocketChannel>() {
#Override
protected void initChannel(SocketChannel socketChannel) throws Exception {
socketChannel
.pipeline()
.addLast(new SimpleChannelInboundHandler<HttpObject>() {
#Override
protected void channelRead0(ChannelHandlerContext channelHandlerContext, HttpObject httpObject) throws Exception {
System.out.println(httpObject);
}
});
}
});
final Channel channel = bootstrap.connect().sync().channel();
final FullHttpRequest request = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/services", Unpooled.EMPTY_BUFFER);
request.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.CLOSE);
channel.writeAndFlush(request);
channel.closeFuture().sync();
System.out.println("DONE");
}
At the moment I am getting
Caused by: java.nio.channels.UnsupportedAddressTypeException: null
Is there an example on how to do HTTP connections to UDS using Netty? So far I only found raw UDS and TCP HTTP but not combined.
Here's a working implementation.
io.netty.bootstrap.Bootstrap bootstrap = new io.netty.bootstrap.Bootstrap();
final EpollEventLoopGroup epollEventLoopGroup = new EpollEventLoopGroup();
try {
bootstrap
.group(epollEventLoopGroup)
.channel(EpollDomainSocketChannel.class)
.handler(new ChannelInitializer<UnixChannel>() {
#Override
public void initChannel(UnixChannel ch) throws Exception {
ch
.pipeline()
.addLast(new HttpClientCodec())
.addLast(new HttpContentDecompressor())
.addLast(new SimpleChannelInboundHandler<HttpObject>() {
private StringBuilder messageBuilder = new StringBuilder();
#Override
public void channelRead0(ChannelHandlerContext ctx, HttpObject msg) throws Exception {
if (msg instanceof HttpContent) {
HttpContent content = (HttpContent) msg;
messageBuilder.append(content.content().toString(StandardCharsets.UTF_8));
if (msg instanceof LastHttpContent) {
System.out.println(messageBuilder);
}
} else {
System.out.println(msg.getClass());
}
}
});
}
});
final Channel channel = bootstrap.connect(new DomainSocketAddress("/var/run/docker.sock")).sync().channel();
final FullHttpRequest request = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/services", Unpooled.EMPTY_BUFFER);
request.headers().set(HttpHeaderNames.HOST, "daemon");
channel.writeAndFlush(request);
channel.closeFuture().sync();
} finally {
epollEventLoopGroup.shutdownGracefully();
}
Few things to note:
Use the EpollEventLoopGroup and EpollDomainSocketChannel with a ChannelInitializer<UnixChannel>.
HTTP requires new HttpCodec() in the pipeline to use the Netty HTTP objects.
The data may be chunked so you need to assemble it and wait for the LastHttpContent object
https://github.com/trajano/netty-docker-daemon-socket
I have a code in C# that access TFS Rest API.
When I use the following code:
VssConnection connection = new VssConnection(orgUrl, new VssCredentials());
I get the results I need.
When trying to set the credentials:
VssConnection connection = new VssConnection(orgUrl, credentials);
I get an error.
An unhandled exception of type 'System.AggregateException' occurred in mscorlib.dll
The error occurs when the following code line is committed:
WorkItem workitem = witClient.GetWorkItemAsync(workItemId).Result;
Full code:
static void Main(string[] args)
{
Uri orgUrl = new Uri("my uri");
string personalAccessToken = "my token";
int workItemId = 486915;
// Create a connection
VssCredentials credentials = new VssBasicCredential("", personalAccessToken);
VssConnection connection = new VssConnection(orgUrl, credentials);
//VssConnection connection = new VssConnection(orgUrl, new VssCredentials());
// Show details a work item
ShowWorkItemDetails(connection, workItemId);
Console.ReadLine();
}
public static void ShowWorkItemDetails(VssConnection connection, int workItemId)
{
// Get an instance of the work item tracking client
WorkItemTrackingHttpClient witClient = connection.GetClient<WorkItemTrackingHttpClient>();
// Get the specified work item
WorkItem workitem = witClient.GetWorkItemAsync(workItemId).Result;
// Output the work item's field values
foreach (var field in workitem.Fields)
{
Console.WriteLine(" {0}: {1}", field.Key, field.Value);
}
}
I used following class which extends the Thread to connect to the web
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import javax.microedition.io.Connector;
import javax.microedition.io.HttpConnection;
import net.rim.device.api.ui.component.Dialog;
public class ConnectJson extends Thread {
private String url;
public String response;
private String myinterface = ";interface=wifi";
public void run() {
HttpConnection conn = null;
InputStream in = null;
int code;
try {
conn = (HttpConnection) Connector.open(this.url + this.myinterface, Connector.READ);
conn.setRequestMethod(HttpConnection.GET);
code = conn.getResponseCode();
if (code == HttpConnection.HTTP_OK) {
in = conn.openInputStream();
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buffer = new byte[in.available()];
int len = 0;
while (-1 != (len = in.read(buffer))) {
out.write(buffer);
}
out.flush();
this.response = new String(out.toByteArray());
if (out != null){
out.close();
}
if (in != null){
in.close();
}
if (conn != null){
conn.close();
}
}
} catch (Exception e) {
Dialog.inform(e.toString());
}
}
public String jsonResult(String url){
this.url = url;
this.start();
this.run();
return response;
}
}
but as you can see the URL is bound to a wifi interface. Look following lines.
private String myinterface = ";interface=wifi";
conn = (HttpConnection) Connector.open(this.url + this.myinterface, Connector.READ);
I want to connect the device to the internet directly via a mobile service provider without using wifi interface. Is it possible to remove this.myinterface if it is possible please tell me how can I do that
I have removed bound to interface and tested it on the device. but it pops out an error saying that
java.io.IOException: APN is not specified
Do I need to specify APN in my code ?
Thanks!