80,443,8080 - Recon

Methodology

• Recon/Scanning

  • Profile Target

  • Identify Web Server Components

  • Test DNS

  • Test username enumeration (via OSINT)

• Vulnerability assessment

  • Test default credentials

  • Test Injections

  • Test for session fixation

• Exploitation

• Post-exploitation

Mindmap

https://github.com/WebBreacher/osinttools

https://twitter.com/ogma_sec/status/1032200415767670784

HTTP/S

HTTP Request - Status Codes

Status Code	Reason
1xx Informational	Request received, continuing process
2xx Success	The action was successfully received, understood, HTTP Response Status Codes and accepted
3xx Redirection	Further action must be taken in order to complete the request
4xx Client Error	The request contains bad syntax or cannot be fulfilled
5xx Server Error	The server failed to fulfill an apparently valid request

HTTP Headers

HTTP Request Example

GET /1/login.php HTTP/1.1
Host: test.com
User-Agent Mozzila/5.0 (x6; Windows) Gecko/20100100 Firefox/67.0
Accept:text/html,application/xhtml+xml
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Referer: https://test.com/
Connection: close
Upgrade-Insecure-Requests: 1

HTTP Response Example

HTTP/1.1 200 OK
Date: Fri, 21 Jun 2000 11:11:11 GMT
Server: Apache/2.4.29 (Ubuntu)
Vary: Accepted-Encoding
Content-Length: 1061
Connection: close
Content-Type: text/html; charset=UTF-8

HTTP Request Methods

Safe or unsafe methods

• Safe Methods: GET, HEAD, OPTIONS

• Unsafe Methods: POST, PUT, DELETE

Repeated submissions of an idempotent method yield same end server state

• Idempotent methods: GET, HEAD, PUT, DELETE, OPTIONS

• Non-idempotent methods: POST, PATH

CRUD methods (common in RESTful web services)

Operation	Method	Notes
Create	POST (also PUT)	PUT less commonly used for creating in RESTful service
Read	GET
Update	POST, PUT, PATCH	In RESTFUL, PUT more ofthen used for replacing resources
Delete	DELETE

Vulnerabilities per available HTTP methods

Knowing the available HTTP methods:

• GET: parameters in teh URI may disclose sensitive information

• Identifies potential vulnerabilities

• Gives clues about how well the system is maintained

• May reveal additional functionality to test

A few examples of how attackers may use Specific HTTP methods:

• GET: paremeters in the URI may disclose sensitive information

• POST: paremeters can be used to exploit vulnerabilities

• PUT: may be able to upload files to the webroot

• TRACE: identify load balancers or other devices in front of the webserver

• DELETE: remove files from the webroot

User-Agent

Used to identify web client. Some tools (sqlmap, nmap) or scripts thjat use Python's urllib3 library may identify themselves via HTTP user-agent. Make sure to customize the user-agent to not allow easy detection.

Referer

Identifies for target server what page the user-agent was viewing when a link was clicked. This can give potential sensitive information being included on the URL.

Cookie and Set-Cookie

Upon sending a Cookie HTTP request header, server will respond with Set-Cookie header. Stealing authenticated cookies can lead to a session hijacking. Secure and HttpOnly flags ensure that transimssion occurs over an HTTPS encrypted channel.

Authorization

Authorization request header is associated with HTTP-based authentication methods. Often times sending Base-64 encoded values via authorization header.

Cacheable and Cache-Control

GET and HEAD requests typically are cacheble. Status codes of 200,203,204,206,300,301,404,405,410,414,501.

If you see a 304: Not Modified, this is because the data was previously cached.

Server

Server field provides HTTP response equivalent to the User-Agent header field. It indicates the web server closest to the end user.

WWW-Authenticate

When server uses built-in authentication method of HTTP Basic or Digest and user authenticates. The server might respond by employing the WWW-Authenticate header.

Structure: WWW-Authenticate: <type> realm=<realm>

<type> identifies the particular authentication method being employed and <realm> describes the "protection space" and indicates the scope of what is protected by this authentication.

Authentication

Web Server Based Authentication

HTTP Basic Authentication

Basic is the simplest built-in authentication scheme defined in RFC 2617. Server sends parameters called Realm, which describes the resource that is protected in in scope. Credentials are stored on the server (.htaccess on Apache). Upon submitting credentials, browser will create a Authorization header, concate credentials with : between and then base 64 encode. Also pads to 3 bytes using the = character.

Issues:

• Without SSL, due to easily decoded base 64 credentials, this can be completely insecure.

• After initial authentication, Authorization header is passed in every subsequent requests (increase attack window)

• There is no log out. Only way to log out is to close browser.

• Depending on server, there is no account lockout

https://dotnetthoughts.net/basic-http-authentication-in-asp-net-web-api/

HTTP Digest Authentication

Designed as a fix to Basic Authentication scheme in RFC 2617, Digest authentication does not send encoded password over the network, but adds additional parameters in challenge-response process. Sets Realm, salts (nonces and cnonces), and md5 to calculate response.

Issues:

• Without SSL, password could still be cracked if nonces and cnonce are captured.

• After initial authentication, the WWW-Authorize header is passed on every subsequent request (Increases attack window)

• No log out (unless browser is closed)

• Depending on server there is no account lockout

https://blog.radware.com/applicationdelivery/2018/05/single-sign-on-sso-use-cases/attachment/basic-digest-authentication/

Tool to crack Digest passwords: https://github.com/eric-conrad/digestive/

digestive --wordlist /opt/john/run/password.lst --username user --realm test --uri /digest-test/ --qop auth --nc 00000001 --method GET --nonce NONCE --response RESPONSE --cnonce CNONCE
# Fill in parameters per captured traffic
# Fill in NONCE, RESPONSE, and CNONCE from captured GET request.

Windows Integrated Authentication

Also known as IWA, is a proprietary authentication schema added by Microsoft. Schema uses NTLM, for NTLM (Challenge-Response protocol) and Kerberos (Handled by client with tickets) over HTTP. Mostly seen in intranets, Sharepoint uses, provides SSO, required both client and server to be on same Windows domain.

NTLM over HTTP is a two round authentication protocol. Can be identified with the beginning string in Negotiate Authorization header T1RMTVNTUAAB. (Base64 encoded string NTLMSSP). Kerberos has only one round also uses WWW-Authenticate: NTLM schema

Issues:

• Without SSL, password could still be cracked if network traffic captured

• After initial authentication, the WWW-Authorize header is passed in every subsequent request (Increases attack window)

• There is no log out. As long as user is logged into domain, browser will automatically authenticate. Opens door to CSRF attack vector.

https://support.okta.com/help/s/article/IWA-Troubleshooting-Guide?language=en_US

Web Application Based Authentication

Form-based Authentication

This is the most common way of user authentication. Developers create HTML form which is used to perform authentication. Submitted credentials are sent in a POST HTTP request (Can also be GET), must be over SSL otherwise credentials are sent in plain text. Normally back-end authentication is used (SQL databases).

Form based authentication can be broken into 3 important components:

• Authentication Form (page that accepts credentials and submits to server-side processing code)

• Processing Code (Server-side code that verifies credentials, upon success redirects user to target page)

• Protected Resources (Any resources that can be accessed only by authenticated users, needed to be configured correctly by devs)

Issues:

• Secured by dev (Without proper security controls, back-end database could be open to injection attacks: SQL, LDAP, etc)

• Session handled by developer or framework (Typically via cookies) (Session handling needs to ensure log out after inactivity or when logout button is pressed)

• Account lockout depends on dev

• Susceptible to HTML injection attacks (XSS)

• Commonly used in phishing attacks

https://docs.oracle.com/cd/E19798-01/821-1841/bncbq/index.html

OpenId/OAuth/SAML Authentication

Allows authentication of users without asking for credentials as authentication is handled by identity provider (3P server). Based on tokens and uses APIs and can allow for SSO.

.

https://www.quora.com/What-does-Quora-do-to-make-Google-and-Facebook-users-login-automatically-Is-this-a-privacy-violation

DNS

Get CIDR of target (Can help with reverse DNS Scanning)

# Due to GDPR might not get as much data from whois
dig test.com
whois 10.10.10.10 # IP collected from dig
# Will possibly output CIDR

Capture all records under a domain

dig test.com -t any

Zone transfer (3 methods)

dig test.com -t axfr # Attempt a zone transfer
nmap --script=dns-zone-transfer test.com # Attempt a zone transfer
dnsrecon.py -a -d test.com # Attempt a Zone transfer

Incremental zone transfer

dig test.com -t axfr
# Capture SOA serial number (e.g 2020011201, YYYYMMDDNN)
# Decrease serial number by 1, and the entire zone will be downloaded
dig test.com -t ixfr=20200011200

DNS brute force (Uses wordlist) for domain names (2 methods)

# Nmap
nmap test.com --script=dns-brute
nmap test.com --script=dns-brute --script-args=dns-brute.hostlist=/opt/dnsrecon/namelist.txt
## By default nmap dictionary has 120+ lines, dnsrecon has 1900+.

# DNSrecon
dnsrecon.py -d test.com -t brt -n 10.10.10.10 -D /opt/dnsrecon/namelist.txt 
## -n is used to give a domain server IP, if none is given the SOA will be used of domain.
## Larger dictionaries can be found:
## /opt/dnsrecon/subdomains-top1mil-5000.txt (5000 entries)
## /opt/dnsrecon/subdomains-top1mil-20000.txt (20000 entries)

Simplified PTR (reverse) lookup

dig -x $IP

Query nameserver's version of BIND

dig @$IP version.bind chaos txt

Reverse DNS Scan to discover hidden CNAMEs (3 methods)

# Use target's CIDR
# Nmap
nmap -sL 10.10.10.0/24 | grep \)

# DNSrecon
dnsrecon.py -r 10.10.10.0/24

# Metasploit
use auxiliary/gather/enum_dns 
set DOMAIN test.com
set ENUM_RVL true
set IPRANGE 10.0.0.0/24
run

Zone transfer using to find internal records

dig axfr @n1.example.com internal

Useful metasploit DNS information-gathering auxiliary modules

• auxiliary/gather/dns_bruteforce

  • Performs brute force dictionary DNS scan

• auxiliary/gather/dns_cache_scraper

  • Queries DNS cache for previously resolved names

• auxiliary/gather/dns_info

  • Gatheres general DNS information

• auxiliary/gather/dns_reverse_lookup

  • Performs a reverse DNS (PTR) scan of a netblock

• auxiliary/gather/dns_srv_enum

  • Enumerates SRV (Server) records

OSINT

Any data collecting where you are not directly scanning an endpoint, but external data collecting.

Search Engines Dorking

What to look for?

• Hardcoded Credentials

• Dependencies

• Versions

• Cached information (Google cache or archive.org)

• Sensitive files

Dictionary Cheat Sheet:

• Google Hacking Database: https://www.exploit-db.com/google-hacking-database

• Bing's Cheat Sheet: https://help.bing.microsoft.com/#apex/18/en-US/10001/-1

• Google's Cheat Sheet: https://support.google.com/websearch/answer/2466433?hl=en

• DuckDuckGo's Ceat Sheet: https://help.duckduckgo.com/duckduckgo-help-pages/results/syntax/

General Search Operators	Meaning
-	Exclude elements
"Server credentials"	Exact match
site:test.com	Filter to only particular site
cache:	Search cache rather than live index
ext: pdf filetype: pdf	Filter particular extension/filetype
intitle: admin inurl: admin	Searches on web page title or url rather than content

DuckDuckGo Specific	Meaning
!	Allows searching based on provider: https://duckduckgo.com/bang (e.g. !a searches amazon, !w wikipedia, etc)

Bing specific	Meaning
ip:1.1.1.1	Finds sites that are hosted by a specific IP address.
prefer:	Adds emphasis to a search term or another operator to help focus the search results.
url:	Checks whether the listed domain or web address is in the Bing index.
tesla near:10 CEO	Searches words near each other
contains:pdf	Keeps results focused on sites that have links to the file types that you specify.

Cheat sheet

Look for Directory Browsering via Google

intitle:"index of" "last modified" site.test.com

Cached content

Manual testing

• Google's cached pages: cache: test.com

• Wayback Machine: https://archive.org/web/

Automated testing

• waybackurl tool: https://github.com/tomnomnom/waybackurls

Shodan

ShodanShodan

Shodan is a search engine that enables users to find specific types of devices connected to the internet.

Unlike traditional search engines such as Google or Bing, that are designed to search the Web, Shodan scans and indexes IP addresses, non-HTTP/HTTPS ports, servers, routers, IoT devices, printers and any physical system connected on the internet.

What can be found:

• Routers

• Webcams

• Power Plants

• Refrigerators

• VoIP Phones

• etc

Netcraft

What's that site running?What's that site running? | Netcraft

Similar to Shodan

Censys

Home - CensysCensys

Similar to Shodan

Social Media

LinkedIn or Twitter

Metadata

Use tool to extract following metadata of files (e.g, pdf, images, etc.) This data can later be used to create wordlist or get additional information of a target:

• Geolocation data

• Software (+version)

• Usernames

• Full File paths

• Code comments

Tools: Tesseract, exiftool, Foca

Virtual Host Discovery

Certificate Transparency reports offer a penetration tester an additional method for discovering hosts and subdomains within a domain. This includes "hidden" virtual hosts that are not linked publicly, but have been issued an x.509 cert.

Use following sites:

• https://crt.sh/

• https://transparencyreport.google.com/https/certificates

crt.sh search for sans.org

Target Profiling

Collect data on what technology supports the web application.

Identify Web Server Components

Webserver Commands	Examples	Common Discovery Path(s)
Web Servers	Apache ISS NGINX	Port Scans, default web pages, fingerprinting tools
Application Frameworks	Spring ASP.NET Django Symfony	Default web pages, vulnerability scans, config files, admi pages, fingerprinting tools
Content Management Systems (CMS)	WordPress Drupal Joomla SharePoint	Default web pages, vulnerability scans, config files, admin pages, fingerprinting tools
Databases	MySQL Microsoft SQL Oracle Postgres MongoDB	Port Scans, detailed application errors, config files
Other Software	OpenSSH RDP SMB	Port Scans

Capture HTTP Header

# Curl
curl -s --head test.com

# Netcat
nc -C test.com 80
nc -v test.com 80
## Get Server info for 1.0
HEAD / HTTP/1.0
## Get Server info for 1.1 (Virtual Host)
HEAD / HTTP/1.1
Host: test.com

# OpenSSL HTTPS
openssl s_client -crlf -connect www.test.com:443
HEAD / HTTP/1.1 # or GET / HTTP/1.0
Host:test.com

# Nmap
nmap -p 80,443,8080 --script=http-headers test.com

Using Nmap Script to capture HTTP Service Information

nmap -p 80,443,8080 --script=http-title $IP #Gather page titles from HTTP services
nmap -p 80,443,8080 --script=http-headers $IP #Get HTTP headers of web services
nmap -p 80,443,8080 --script=http-enum $IP #Find web apps from known paths

Testing URI query vs POST parameters

Sensitive data being set via URI query is a finding. Data sent HTTP body via POST request can be encoded/encrypted

Test for Verb Tampering and GET/POST Method Interchange

• Use Burps "Change request method" to see if application allows swapping. This is a finding.

Test for TRACE method

TRACE method allows client to see what is being received at the other end of the request chain and use that data for testing or diagnostic information

curl -i -H "Cookie: answer=42; question=darthvader" -X TRACE https://test.com

Test with OPTIONS method

OPTIONS method allows for asking servers what methods they support. Make sure to test for methods not included.

Test for Server field included in HTTP response

curl -s --head test.com | grep -i server

# For multiple domains
curl -s --head -K servers.txt | grep -i server

Checking for Security Headers

• Use burp/zap as interpreter and analyze headers

• Use browser add-on Wapalyzer which analyzes headers: https://www.wappalyzer.com/

• Good site to analyze HTTP Headers of external facing sites: https://securityheaders.com/

Analyze cipher strength of HTTPS SSL

External sites:

• Qualys SSL Labs: https://www.ssllabs.com/ssltest/

• Security Headers: https://securityheaders.com/

nmap --script=ssl-enum-ciphers -p 443 $IP

You can also manually run TestSSLServer.jar: https://github.com/timbo05sec/Tools-n-Stuff/tree/master/4_exploitation (Must use old java as older SSL versions are deprecated on newer Java versions)

Username Harvesting

Per HTML response/visible response

• Typing a username, is it visible if its valid or does respond return username value in response?

  • We can use bash script or burp to send a wordlist as username and random password and based on length of response, it will identify what are valid users.

Timing techniques without HTML differences

• Knowing username formatting (e.g 1 Letter for first name and full last name), create word dictionary of top 100 last names and run a Fuzzer against login and sort response by RTT

Versioning Port Scan

sudo nmap -sV test.com

# or bash script it
for port in (0..25565}
    do
        nc -vz $IP $port 2>&1 | grep suceeded
done

Look at robots.txt

curl http://www.test.com/robots.txt
nmap --script=http-robots.txt test.com

Finding Hidden parameters (Arjun)

python3 arjun.py -u http://$IP/phpMyAdmin/

# Search for GET parameters
python3 arjun.py -u http://$IP/phpMyAdmin/ --get

# Search for POST parameters
python3 arjun.py -u http://$IP/phpMyAdmin/ --post

# Search for JSON parameters
python3 arjun.py -u http://$IP/phpMyAdmin/ --json

# Search forusing own parameter wordlist
# Example: https://github.com/Damian89/xssfinder/blob/master/wordlists/params.txt
python3 arjun.py -u http://$IP/phpMyAdmin/ -f params.txt

# Search parameters with specific header
python3 arjun.py -u http://$IP/phpMyAdmin/ --headers

AJAX Applications

--

Automated Tools

OSINT

Data mine tools:

• Maltego (Commercial use)

• SpiderFoot (OpenSource)

Warning: May break Privacy Terms for Pentest

Searching domain names and email addresses:

• theHarvester

• AutOSINT (Uses theHarvester and pyFOCA to get OSINT data)

Spidering

• Burp Spider (Commercial)

• Zap Spider

• gospider

gospider -S tageturls.txt -c 10 -d 5 --blacklist ".(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico|pdf|svg|txt)" --other-source | grep -e "code-200" | awk '{print $5}'|grep "=" | qsreplace -a | dalfox pipe -o result.txt

Alternative methods to spidering for Single Page Application (SPA) or AJAX applications

• Use an AJAX spider (not reliable)

• Manual clicks + Proxy

Forced Browsing (Directory Brute Forcing)

• dirbuster (Old)

• dirb

• gobuster

• curl

• Nikto

• w3af

• Metasploit's WMAP and msfcrawler auxiliary module

#Single line Force Browsing with curl and dirb wordlist
while read dir; do curl -Is http://domain.test.com/$dir | grep -E '200 |302 ' && echo $dir; done < /opt/dirb/wordlists/big.txt

Force Browsing dictionaries

• SecLists

• Dirb

• DirBuster

• FuzzDB

• JBroFuzz

• WMAP

Get Subject Alternative Names (SAN) Certificate

openssl s_client -connect website.com:443 | openssl x509 -noout -text | grep DNS:

Download and use client SSL certificate

openssl s_client -showcerts -connect website.com:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >mycertfile.pem
wget https:/server.edu:443/somepage --ca-certificate=mycertfile.pem

Show SSL certification

openssl s_client -connect website.com:443 -showcerts

Fuzzing

Fuzzing involves replacing normal values with attempted exploits to all available inputs and reviewing responses

What needs to be fuzzed

• Request headers

• POST parameters

• GET parameters

• PUT payloads

• Any input to client-side and server-side code

Input Examples

• SQL Injection ('+or+1=1;%23)

• Password Spraying(Different usernames same password)

• XSS (<script>alert("42")</script>)

• Directory Traversal / Local File inclusion (../../../../../etc/passwrd)

Fuzzing Dictionaries

• SecLists: https://github.com/danielmiessler/seclists

  • Download to /opt/seclists for use

• Google's Fuzzing dictionaries https://github.com/google/fuzzing/tree/master/dictionaries

• ameenmaali's qsfuzz (Query String Fuzz) allows you to build your own rules to fuzz query strings and easily identify vulnerabilities. https://github.com/ameenmaali/qsfuzz

Fuzzing Tools

• ZAP's Fuzzer

• patator

• ffuf

• Wfuzz

VHOST Discovery

gobuster  vhost -u test.com -w ~/SecLists/Discovery/DNS/subdomains-top1million-110000.txt

SAST

Strenghts: Identifies Security Deficiencies not readily apparent in deployed applications

Weakness: Requires access to source code; might overlook APIs or libraries leveraged by the application; overlooks ops side of apps

DAST

Strengths: Doesn't require source code access; necessarily accounts for ops side of apps; can be automated/integrated into pipeline; scales well for large applications

Weaknesses: Only as good as the tool, its configuration, and the user wielding it; significant incidence of false positive; difficulty with certain types of flaws

Tools

Free/Open Source	Commercial
ZAP Active Scan	Acunetix Vulnerability Scanner
SQLmap (SQL Injection specific)	Burp Scanner
W3AF	Fortify WebInspect
Metasploit WMAP	IBM AppScan
WPScan (WordPress specific)	Qualys WAS
	Rapid7 AppSpider
	Veracode Dynamic Analysis
	Whitehat Sentinel
	ZAP Scan

LinkFinder

LinkFinder is an open-source python script used to discover API endpoints, URL resources and query parameters within minified JavaScript files. The script uses a combination of regular expressions to gather hidden URLs and application routes which can then be used by security researchers to further test for vulnerabilities.

Example:

python linkfinder.py -i https://google.com -d -o cli

#i: Input a: URL, file or folder.
#d: Analyzing an entire domain and its JS files
#o cli: CLI/STDOUT output

Other Resources

Web App Hacking, Part 15: Web site Fingerprinting with Whatweb https://www.hackers-arise.com/post/2019/05/14/web-app-hacking-part-13-web-site-fingerprinting-and-vulnerability-scan

Arujin - Discover Hidden HTTP Parameters with Arjun https://null-byte.wonderhowto.com/how-to/discover-hidden-http-parameters-with-arjun-0212025/

Create HTTP/s whitenoise in upstream network logs/taps: https://github.com/Relkci/ps-whitenoiseweb

Abusing HTTP Path Normalization and Cache Poisoning to steal Rocket League Accounts: https://samcurry.net/abusing-http-path-normalization-and-cache-poisoning-to-steal-rocket-league-accounts/