NOTICE: Ciuffy will be answering questions related to my articles on my behalf as I am very busy. Hope You Have Fun !!!
Hello Guys,
Welcome to my tutorial and in this tutorial, We are basically going to
create a python script that serves as a proxy server ( Without External Libraries ).
I made this script when I had some knowledge of python sockets and also thought it was a cool thing to do. I want to share it with the community.
Before I begin, I would first like to thank the members who gave me positive feedbacks on my How To Make A Python Scanner Article, I am very grateful for your appreciation.
Also, Lets quickly talk about Proxies before we proceed
any further to help others who might have heard or read the name
somewhere but don't have a slightest idea of what it means.
INTRODUCTION INTO THE WORLD OF PROXIES
From WikiPedia - Proxy Server.
In computer networks, A proxy server is a server ( A computer system or
an application ) that acts as an intermediary for requests from clients
seeking resources from other servers. A client connects to the proxy
server, requesting some service, such as a file, connection, web page,
or other resource available from a different server and the proxy server
evaluates the request as a way to simplify and control its complexity.
Proxies were invented to add structure and encapsulation to distributed
systems. Today, most proxies are web proxies, facilitating access to
content on the World Wide Web and providing anonymity.
Lets see an illustration.
What Bob thinks is the server ( i.e the proxy ) asked for the current time, But what Bob didn't know was, Alice asked for the current time but through the proxy server. The proxy server returns the current time to Alice. So we can basically say, Server Bob has been tricked. The proxy server acts as a man in the middle serving two people without revealing their identities to each other, Each person sees only the proxy but not the other end.
While a proxy might seem straightforward to the typical end user, there are many variations of the simple proxy website. While you might want only you chat with Facebook friends while at work and need to hop online anonymously through a proxy, what's happening behind the scenes can be quite complex. While many proxies look similar on the user's end, they can be very different to the developers and programmers.
Note: There Are Many And Countless Ways To Detect Proxy Clients
There are many types of proxies of which I won't be listing. Here are the most basic and commonly used ones.
TYPES OF PROXIES
From ProxySites and WikiPedia:
PHP Proxy – A PHP proxy uses a script written in PHP to take webpages from your regular server and then processes your search request to protect your main server. A PHP proxy can be a form of CGI proxy or simply a bit of code to attach to the URL as you type it in. PHP proxies can also be used by site and program developers to help cloak their original server to increase the effectiveness of web applications.
CGI Proxy – A CGI proxy allows you to search through the proxy site for the URL you'd like to reach. CGI proxies allow you to search anonymously by hiding your IP address behind the proxy site. Thanks to this simplicity and anonymousness, CGI proxies are the most common proxies for the typical internet user searching for a means to unblock a website or search anonymously.
HTTP Proxy – The HTTP proxy works with web browsers (HTTP clients) and servers that support HTTP. The HTTP proxy caches pages, or store them, for faster retrieval and are used occasionally to increase metadata services available on the website.
Gateway Proxy – A proxy that allows requests and information to travel through without modification is called a gateway, or gateway proxy. These can also be called tunneling proxies as they "tunnel" around blocks put into systems through filters. Tunneling proxies are used frequently in workplaces and educational offices where there are frequent blocks on material that employees or students wish to access.
Content Filter – The filter that might be blocking the sites you wish to access is likely a form of a proxy as well. These specialized proxies called content filters require all requests to pass through for review. The content filter proxy then blocks requests that match requests on a "black list" or based on certain security settings. Depending on the complexity of the content filter, network administrators can add to or remove items from the black list.
DNS Proxy - A DNS proxy server takes DNS queries from a ( Usually local ) networks and forwards them to an Internet Domain Name Server. It may also cache DNS records.
Anonymous HTTPS proxy - Users wanting to bypass web filtering, that want to prevent anyone from monitoring what they are doing, will typically search the internet for an open and anonymous HTTPS transparent proxy. They will then program their browser to proxy all requests through the web filter to this anonymous proxy. Those requests will be encrypted with https. The web filter cannot distinguish these transactions from, say, a legitimate access to a financial website. Thus, content filters are only effective against unsophisticated users.
Use of HTTPS proxies are detectable even without examining the encrypted data, based simply on firewall monitoring of addresses for frequency of use and bandwidth usage. If a massive amount of data is being directed through an address that is within an ISP address range such as Comcast, it is likely a home-operated proxy server. Either the single address or the entire ISP address range is then blocked at the firewall to prevent further connections.
Suffix proxy - A suffix proxy allows a user to access web content by appending the name of the proxy server to the URL of the requested content. Suffix proxy servers are easier to use than regular proxy servers but they do not offer high levels of anonymity and their primary use is for bypassing web filters. However, this is rarely used due to more advanced web filters.
TOR Proxy - Tor ( The Onion Router) is a system intended to enable online anonymity. Tor client software routes Internet traffic through a worldwide volunteer network of servers in order to conceal a user's location or usage from someone conducting network surveillance or traffic analysis. Using Tor makes it more difficult to trace Internet activity, including "visits to Web sites, online posts, instant messages and other communication forms", back to the user. It is intended to protect users' personal freedom, privacy, and ability to conduct confidential business by keeping their internet activities from being monitored.
"Onion routing" refers to the layered nature of the encryption service: The original data are encrypted and re-encrypted multiple times, then sent through successive Tor relays, each one of which decrypts a "layer" of encryption before passing the data on to the next relay and ultimately the destination. This reduces the possibility of the original data being unscrambled or understood in transit.
I2P anonymous proxy - The I2P anonymous network ( I2P ) is a proxy network aiming at online anonymity. It implements garlic routing, which is an enhancement of Tor's onion routing. I2P is fully distributed and works by encrypting all communications in various layers and relaying them through a network of routers run by volunteers in various locations. By keeping the source of the information hidden, I2P offers censorship resistance. The goals of I2P are to protect users' personal freedom, privacy, and ability to conduct confidential business.
Each user of I2P runs an I2P router on their computer (node). The I2P router takes care of finding other peers and building anonymizing tunnels through them. I2P provides proxies for all protocols (HTTP, IRC, SOCKS, ...).
SOME USES OF PROXIES
We may all have our reasons for using proxies, But mostly, People use proxies for anonymous surfing, hacking D-Grade websites and playing other pranks on the web. Most common uses are as follows;
- Filtering of encrypted data
- Bypassing filters and censorship
- Logging and eavesdropping
- Improving performance
- Security
- Cross-domain resources
- Translation
- Anonymousity
Ok, I think that is enough for a brief introduction.
BUILDING OUR PROXY SCRIPT
The script is cross platform since the modules imported are cross-platform i.e available in both windows python and linux python.
SCREENSHOT 1: IMPORTING MODULES
import socket, sys - Import the socket module, Import the sys module. ( We can import as many modules as we want so far as we put a comma in between them )
from thread import * - From the thread module, Import everything ( functions )
SCREENSHOT 2: DECLARING PROGRAM SETTINGS
listening_port - We ask the user for a listening port using the raw_input function, After we assign the input to the listening_port variable.
max_conn - We limit the amount of connections to our proxy server.
buffer_size - Maximum socket buffer size ( Recommended: 4096 )
Except KeyboardInterrupt
This handles keyboard interrupts ( Ctrl + C ) should the user want to close the script for whatever purposes. This prevents python from stopping our program execution and spilling its Keyboard Interrupt exception code to the screen. ( Kind-off seems unprofessional ). Like this ..
Let's see what our program does when a user hits the interrupt command.
Nicely done !!!
SCREENSHOT 3: INITIATING THE PROGRAM
We are going to enclose all our codes in functions and later call them during our usage.
We are going to use three functions in the script.
Function 1: start() - Main Program
Function 2: conn_string() - Retrieve Connection Details
Function 3: proxy_server() - Make connection to end server
SCREENSHOT 4: Function: start()
What this function does is to
1. Set up the socket
2. Bind the sockets
3. Start listening for incoming connections
When and error occurs, Print the custom message and exit the script
4. When a connection arrives from the client, accept it, receive the clients data ( Browser request ) and start a thread. The thread in-turns calls another function conn_string and pass three arguements:
- conn ( Connection Stream )
- data - clients data ( Browser request )
- address - ( Address from the connection )
When a user interrupts the program, the server connection stream closes, prints the custom message and exits the proxy script.
SCREENSHOT 5: Function: conn_string()
Arguements: conn, data, addr
This is an example of a client browser request ( The connection is to Null Byte ).
What this function conn_string basically does is to retrieve the host address, port (if specified by client) in the received client data, call another function proxy_server and pass five arguements to it.
webserver - host address
port - webserver port
conn - connection stream
addr - address from connection stream.
data - client browser data
SCREENSHOT 6: Function: proxy_server()
Arguements: webserver, port, conn, data, addr
NOTE: An error I made. The s.connect function in the fourth line in the image below should have the variable webserver instead of host. Failure to change will deny the user of relaying the information to the internet.
This function creates a new socket, connects to the webserver and sends the clients request, receives the reply, forwards it back to the client without modifying it, prints a custom success message and closes the streams.
SCREENSHOT 5: FINAL WORK
Let's make the magic work ... We simply type start() on a last line of the script.
What this does is to start the main program .... We now have a working python script .....
Not soooo fast ??? We haven't tested it. ( A connection to google )
To use the script, Simply set your browser proxy to 127.0.0.1 and desired port, Make sure your script is running on that port ( Mine is on port 8001 )
Test Run.
Ahh !! :) ......... Successfully working.
CONCLUSION
I hope we had fun and learnt something today. This script can be hosted online ( maybe on a vps ) to serve as a proxy and receive requests from clients. We can also make logs from the script right after sending the reply back to the client.
Notify me of any errors, misinformation or anything that needs correction. Hope to see you soon and Have a nice day !!!
# Sergeant
Just updated your iPhone? You'll find new emoji, enhanced security, podcast transcripts, Apple Cash virtual numbers, and other useful features. There are even new additions hidden within Safari. Find out what's new and changed on your iPhone with the iOS 17.4 update.
63 Comments
Very fun guide, I always love when we create something rather than use something pre-existing.
ghost_
Who's sniping my comments again?!
Anyway, I said what ghost_ said sounded like one of my other comments.
What?
ghost_
I replied to your comment that it sounded like what I said on Ciuffy's post. "It's no fun using someone else's tool."
My comment was deleted, however. I don't know why, but it's not the first time. My comments were sniped a lot a few months ago.
Think the DB needs some love or somebody is in the back end because I'm not getting notices either.
Your guides are really detailed, and I agree with ghost_.
Thanks Guys ... Really appreciate
# Sergeant
I keep getting 'Unable to Initialise Socket' -- I have audited the code several times -- no errors to speak of, however I notice that your start() is at line 145, whereas mine is at 106? Insight on this would be appreciated. :v
Elliot:
Unable to Initialize Socket - The only problem could be your port number, It may be in use. Just change it with a different port i.e number.
# Sergeant
Thanks for your reply,
I've tried numerous port#s, with the same result. ??
When I load my .py, I get this prior to run, not sure if it has anything to do with the issue.
usr/share/pype/plugins/spellcheck.py:35: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal nonword.update(dict((unichr(i), ' ') for i in xrange(256) if chr(i) not in goodch)) --
err.. is this typical? lol.
Am I missing a value, between ' ' ? :I lol
Elliot:
I don't get the origin of the error output either. Please post your code i.e in 3 different screenshots. That way i can help.
( Sorry for late reply )
# Sergeant
Thanks again for the tutorial, and your help. (;
I should note: the 8001 was added afterward when I began receiving the Unicode error ; everything was as it should be prior to. :P
It seems like I am missing some 40 lines of code, as I mentioned earlier -- but aside from the 'get/host/user-agent' block, I can't see what I might be missing.
When does the error occur, immediately after program start up or during an input or ... ?
A screenshot of the error also
# Sergeant
Funny. When I was looking at your first screenshot, I noticed:
Hope it helps
# Sergeant
Awesome!!!!
Thanks friend :)
# Sergeant
Hey Sergeant Sploit!
Good tutorial !
So far i try to understand every line of code, but it's still don't working :/
the script is running (server started etc.) but i don't receive any data.
Iceweasel returns an "... proxy refused connection".
i think i might missed some of the code, because the 'start()' is in line 145 in your script, in mine its line 106 :0
sry for my bad english
greetings m0
Don't mind my line numbers, I had to space out the codes in order to have a full and clear screenshot. If you have any problem, You can output your error and also your code to see if anything was missed.
( A more detailed description of your problem may do along with some screenshot )
# Sergeant
## --- He may have comment notes in the script that would space it out as well.
My random input.
Connection refused:
The peer is not listening on the port you specified.
Firewall is blocking
Service is not started.
187 other reasons.
I rarely call for the screen but we are gonna need a screeners of the code and the output please.
Do you only get this error when using a browser to bounce?
Thanks for your answer ! :)
here are some screens,
i'm looking through the code but cant figure out whats wrong :/
i only used Iceweasel so far.. i ran the script, choose a port (screen), started iceweasel and set proxy conf (screen). but it seems that no data is going through... because connection was reset.
m0
Funny, Lolx, Take a look @ your second screenshot.
And see my second screenshot.
After temp = url(http_pos+3):, the rest of the codes are not under the else statement, That is why the script cannot retrieve the connection details as well as making a connection because we passed all errors so you couldn't see it.
And it seems you messed the codes big time.
NOTICE: The First Screenshot: I closed the socket "s.close()" before exiting the script"sys.exit()"
# Sergeant
Hey Sergeant !
Thanks for your reply !
I changed the code like you did in your second screenshot.
I'm still trying, but can't get i to work :/
what do you exactly mean with "And it seems you messed the codes big time." ?
sry i'm still learning :)
greets m0
Watch my screenshot and compare if you missed anything or misplaced a code.
( Still can't find, Post your screenshot )
# Sergeant
Might be more easy if you just start again from line 1 and rewrite it Exactly the way he has it coded.
Your indentation is horrible to start off with, no disrespect.
Hey Sergeant
...Awesome tutorial very detailed i pretty much did everything correctly gone over the code a lot of times...but my problem is I can connect to http websites correctly that is sites with port 80 but i can't connect to https websites(443) if i try connecting to say youtube i get a "Secure Connection failed!" . I hope you can help.. thanks
Wuzi:
I checked your code and didn't find anything wrong, I also haven't come across your problem so what I recommend is to edit your script to print any error message. That should enable debug the problem.
( I may not be able to reply in time so hope you understand )
# Sergeant
How to solve this 'Secure Connection Failed' error???
Hey Sergeant, I am having difficulties running my python server. In my browser, i get connection refused.
( I have been following your tutorials from facebook and I must say: "They are really impressive" ). Great tutorial by the way.
Thanks a lot Sergeant. And I do understand thanks for the reply though.
And @Adams I think you may either have used a different proxy port for the script and the browser or the proxy server isn't running as well. So i advise that you check to make sure the ports are the same in the script and the browser settings and also the proxy server is running.
I don't know but I can't just get this thing to work? Any help :(
Post the code you are using on pastebin and show it to us.
Will try when i get home ...
i am a new (kali) user following all the basic tutorials on this site. I am trying to connect to the internet via my mobile subscription (blackberry) on my previous windows installation all i do is run a simple server program which executes the contents of a .ini file and change my proxy settings to 127.0.0.1:8080
please is there any way to create such a program or script on linux.... i would really appreciate any help with that. thanks :)
i have attached a photo of it here to clarify my question.... here
the windows executable of the program and ini file are here
Let me understand better:
1) Are you asking for a script that enables you to route all the traffic trough a proxy which was previously setup, or how to build the proxy itself?
2) What's the blackberry subscription about? Are you trying to setup a proxy for the whole LAN?
3) Iptables? Proxy chains?
how to build the proxy and a script routing all traffic through it....
the proxy enables the machine utilize the connections of the blackberry subscription.
You may want to try the script in python explain in this tutorial and combine it with proxychains or Iptables redirect.
Nice tut
Here's a hint to where to look:
After line (should be) 43
url = first_line.split(' ')[1]
you can get the method (GET, POST, HEAD...) with
method = first_line.split(' ')[0]
After you made the check, remember that the full data is passed to function proxy_server() around line 73 so you should operate a replace on the first bytes of data to get what you want.
Nice tutorial right there, I was wondering if you could be able to tweak each and every CONNECT request to POST method and ignore or don't change all the others that are listening from the made server. Like for instance if an OPENVPN Client (or any client) is using http-proxy 127.0.0.1:this proxy's listening port when connecting to it's desired server, be changed from using connect to use post....
e.g a request like this:
CONNECT xxxxx:443 HTTP/1.0
User-Agent: Mozilla...
http-proxy:127.0.0.1:this proxy's lport
to:
POST xxxxx:443 HTTP/1.1
User-Agent: Mozilla...
http-proxy:127.0.0.1:this proxy's lport
How could the if and else statements look like on this tut while trying to achieve what I said?
I must have pushed the wrong reply button lol... I answered your question to the comment right above yours.
Anyway, also look into twisted, it already has all you need for handling proxy connection, add / modify headers, ...lots of interesting things.
very great tutorial and many thanks. It is working as expected but I have a quick ? If I wish to send another data request via the first port under start(), I can not see the data under def proxy server i.e. if ALice ask again what time it is? The only way I can make it happen is by creating a new socket from the perspective of Alice? It seems just stuck in while loop under proxy server. Just waiting for any data received from the other servers? Hope this makes sense. thanks for any feedback.
What do we have to type in python for the keyboard interrupt part?
Ctrl + C
In def proxyserver I thing there is no variable host...Maybe this should be replaced by the webserver variable?? (line 89 for me)
Thank you IMISSTHESHORE. I noticed it yesterday as I was trying out a project for my free week. I have reflected it in the text above that image.
# Sergeant
I've changed some of the code mainly using select and the re module. I can't seem to get it working if you can help or, anyone else let me know and we can github this. :o
email: yuri9911324@gmail.com
code: http://pastebin.com/psgPAgFV
email in the pastebin post is being ew atm
please note this only runs on py2.7 I haven't made it cross compatible and there prints where returns should be for debugging purposes
Hello Michael, I just run through your code ( actually I am not looking into it because I don't want to dive into any computer based stuff now ) and noticed an error like print(" Request Done: %s => %s <= %s" % (self._addr0, dar))_ on line 99. All what I would like you to do is to take your time and match references for the new modules you are using and also make sure you debug each stage of a function. The error on line 99: %s are 3 whilst the % ( the variable reference symbol ) is only addressing 2. Please do take your time and look through the code. Should the problem still persist, you can contact the administrator, I think he knows some python. Also you cannot modify HTTPS connections, only HTTP. Modifying HTTPS is possible but not possibly with this code. Hope you understand me.
# Sergeant
I noticed that in screenshot 5 the call to proxy server passes addr, data as the last two arguments. However in screenshot 6, the parameters of the proxy server function accept data FIRST followed by addr. Is that supposed to happen?
You're right. There's an error in author's code. The last 2 args in function proxy_server() definition have to be swapped. So, if we leave the call statement unchanged - the definition has to look like :
def proxy_server(webserver, port, conn, addr, data)
Otherwise we proxy out the address, not request.
Great tutorial but i still have some problems.
When i start the script is this the result:
* Enter Listening Port Number: 8080
* initializing Sockets ... Done
* Sockets Binded Successfully ...
* Server Started Successfully 8080
Traceback (most recent call last):
File "/xxxxx/Schreibtisch/proxytest.py", line 105, in <module>
start()
File "/xxxxx/Schreibtisch/proxytest.py", line 31, in start
conn, addr = s.accept() #Accept connection from Client Browser
File "/usr/lib/python2.7/socket.py", line 202, in accept
sock, addr = self.sock.accept()
error: Errno 22 Invalid argument
Thank you in advance for soure help
>>>
Hello Nicht,
From the look of your error, it seems to originate from the server accepting client connection. Please do post a screenshot of that section of your code and let me have a look at it.
# Sergeant
Can anyone post the whole working code in pastebin ?
Hi, love your code.
I would like to use it to build a proxy circumventing csrf protection and publish it (with creds to you) on github. Is this ok for you?
When I try execute:
s.connect((webserver, port))
s.send(self.data)
I get exception:
getaddrinfo faild
This problem exists, because I don't resolve dns name?
Sorry My English!)
Thank you for an awesome tutorial
After finding and correcting a number of tiny errors in my code I am still getting 'Unable to initialise socket'
It may be that I have misunderstood your last suggestions about changing port settings - is this somethng I should do in my setting preferences (I have a Mac)?
Hope you can help.
Mo
Can you please upload a tutorial on how to set this proxy_server .py script on a vps.
hi please can you take a look at my code as i cant see what is wrong
_I got a couple of questions, can it be used on udp? And can it be used with ipv4 and ipv6?
the break statement in
if (len(reply) > 0):
highlights in red and won't work with this error i have googled but did not find anything that worked
line 92
else:
^
SyntaxError: invalid syntax
Hi, I copied you code and changed what people say it is wrong (in the proxy server switched data and addr) and even after it did not work I would love if someone could tell me what I did not do right. thank you. I added the place were I changed the proxy settings and what the result is. and the co
de.
did anybody help me out about LEE PRIVATE's problem, I am facing the same problem
Share Your Thoughts