Web Requests
HTTP Fundamentals
HTTP(HyperText Transfer Protocol) is an application-level protocol used to access the World Wide Web resources. HTTP communication consists of a client and a server, where the client requests the server for a resource. The server processes the requests and returns the requested resource. In browser, we enter FQDN(Fully Qualified Domain Name) as a URL(Uniform Resource Locator) to reach the desired website.
The structure of a URL:
http://admin:password@example.com:80/dashboard.php?login=true#status
Scheme http://
User Info admin:password@ This is an optional
component.
Host example.com
Port :80
Path /dashboard.php If no path specified,
returns the default index.html page.
Query String ?login=true
Fragments #status
HTTP Flow:
User --enters the URL--> Browser
Browser --sends a request to resolve the domain--> DNS Server
DNS Server --looks up the IP and returns it--> Browser
Browser --sends a GET request to the default HTTP port, asking for root
path--> Web Server
Web Server --returns an index file--> Browser
// Our browser usually first look up records in the local '/etc/hosts'
file before contact other DNS servers.
One significant drawbacks of HTTP is that all data is transferred in clear-text. So, MitM huh?
HTTPS(HTTP Secure) makes all communications are transferred in an encrypted format. (Use wireshark to see the difference)
HTTPS Flow:
User --enters the URL--> Browser
Browser --sends HTTP request--> Web Server
Web Server --redirects to HTTPS--> Browser
Browser --sends Client Hello with info--> Web Server
Web Server --sends Server Hello and Server Key--> Browser
Browser --verifies the key and sends Client Key(Encrypted Handshake)-->
Web Server
Web Server --sends back Encrypted Handshake--> Browser
--HTTP communication is continued--
HTTP communications mainly consist of HTTP Requests & Responses.
The structure of a HTTP Request:
Method GET
Path /users/login.html
Version HTTP/1.1
Headers Host, User-Agent, Cookie, etc
The structure of a HTTP Response:
Version HTTP/1.1
Response code 200 OK
Headers similar to request
Body HTML code, JSON, etc
HTTP Headers pass information between the client and the server.
There’re different kinds of headers:
- General Headers
- Date: date and time at which the message originated
- Connection: 'close' or 'keep-alive'
- Entity Headers
- Content-Type: type of resource being transferred
- Media-Type: similar to Content-Type
- Boundary: a maker to separate content
- Content-Length: size of the entity being passed
- Content-Encoding: transformations before being passed
- Request Headers
- Host: the host being queried for the resource
- User-Agent: the client requesting resources
- Referer: where the current request is coming from
- Accept: with media types the client can understand
- Cookie: cookie-value pairs
- Authorization: method for the server to identify clients
- Response Headers
- Server: info about the HTTP server
- Set-Cookie: the cookies needed for client identification
- WWW-Authenticate: the type of authentication required
- Security Headers
- Content-Security-Policy: the website's policy towards externally
injected resources
- Strict-Transport-Security: forces all communication to be
carried over HTTPS
- Referrer-Policy: should include referer header or not
HTTP Methods
Request Methods
- GET
- POST
- HEAD
- PUT
- DELETE
- OPTIONS
- PATCH
Response Codes
- 1xx Provides info and does not affect the processing of the request
- 2xx Returned when a request succeeds
- 3xx Returned when the server redirects the client
- 4xx Signifies improper requests from the client
- 5xx Returned when there is some problem with the HTTP server itself
Whenever we visit any URL, browsers default to a GET request to obtain the remote resources hosted at that URL.
Basic Auth
$ curl -u admin:password http://example.com:80/
$ curl http://admin:password@example.com:80/
// use curl -v <target> to check the header
$ curl -H 'Authorization: Basic <base64 encoded admin:password>'
http://example.com:80/
Whenever web applications need to transfer files or move the user parameters from the URL, they utilize POST requests.
$ curl -X POST -d 'username=admin&password=password'
http://example.com:80/
// use -i to view the response, it should contain Set-Cookie header
$ curl -b 'PHPSESSID=<cookie>' http://example.com:80/
$ curl -H 'Cookie: PHPSESSID=<cookie>' http://example.com:80/
// send JSON data
$ curl -X POST -d '{"search":"london"}' -b 'PHPSESSID=<cookie>' -H
'Content-Type: application/json' http://example.com:80/search.php
We can use CRUD API to perform operations.
- Create POST
- Read GET
- Update PUT
- Delete DELETE
examples:
$ curl -X POST http://<SERVER_IP>:<PORT>/api.php/city/ -d
'{"city_name":"HTB_City", "country_name":"HTB"}' -H 'Content-Type:
application/json'
$ curl -s http://<SERVER_IP>:<PORT>/api.php/city/HTB_City | jq
$ curl -X PUT http://<SERVER_IP>:<PORT>/api.php/city/london -d
'{"city_name":"New_HTB_City", "country_name":"HTB"}' -H 'Content-Type:
application/json'
$ curl -X DELETE http://<SERVER_IP>:<PORT>/api.php/city/New_HTB_City