Category Archives: python

infosec miscellany programming python technical

Bing IP Search via Python

Some time ago I wrote a small python script to search and parse out results from an IP: search on Bing. You can get it from the githib repo here. I had to work around some odd/broken behaviour from Bing along the way.

The Bing IP: operator allows you to search for an IP address and return results from any sites sharing that IP. Can be useful, but it was clunky to do manually so I wrote a script to do it all in the console.

I haven’t used it in years but recently thought to dust it off to publish on gitHub, and after searching through some old folders I dug out the most recent version I could find. It needed updating to Python 3 which was a simple matter of updating a few lines to print(), but I noticed some other problems which took longer to debug.

Firstly, the script worked but the result set was really short. After some manual testing I discovered this was due to some apparent bug/faults with Bing itself which have developed since I first wrote the script.

In general, the Bing IP search seems to be quite neglected: if you visit the default search form at bing.com and enter ip:204.79.197.200 – (bing.com) – for example, it returns an empty page (really empty – a blank, 0kb http response body). Removing all but the actual query parameter from the url string causes the page to render, so you end up with a url like this which works:

https://www.bing.com/search?q=ip%3A204.79.197.200

Which is what the script uses.

The main problem however was the truncated results set. The script tries to load more pages but beyond the first page they are empty – even with the parameter stripping hack which works on the first page, the rest are back to an empty response. It turns out that to load additional results pages, the additional URL parameter ‘first’ is required (eg first=11 – start from result 11) and it appears that more than one parameter used with ‘ip:’ alone in the query string breaks the site.

I confirmed this behaviour with a simple test which resulted in an empty response:

https://www.bing.com/search?q=ip%3A204.79.197.200&foo=bar

So it seems something is definitely breaking on the bing backend, and it happens when IP: is used as the query plus additional GET params; however I found that everything worked as expected as long as IP: was not the only search term in the query parameter itself. I still wanted just the results from the ip search without any modifiers/filters, so I tested out some Bing search operators and came up with these workarounds which resulted in all pages loading as advertised:

  • () ip:204.79.197.200
  • ip:204.79.197.200 OR ip:204.79.197.200
  • +ip:204.79.197.200

The () operator means (include these search terms). I left it blank to see what the behaviour was: it seemed to partially work so far as working around the page loading bug. Same with the OR operator, although both of these returned less results than the last one I tried (+) so I stuck with that.

The (+) operator simply means ‘this term must be included’. Seems redundant with only one search term, and I wasn’t sure how it would behave when applied to another search operator, but it worked and returned more results than the previous attempts so I settled on that.

You can check it out on github.

infosec python technical

Parsing OpenVAS reports in Python

I was using OpenVAS to do some network auditing and accessing report results via the (Greenbone Security assistant) web interface quite often seemed somewhat slow and clunky. The report is downloadable as an XML file though, and I’ve recently been getting familiar with parsing nmap XML files in python, so a bit of scripting later and voila! GOXParse (Glens OpenVAS XML Parser) – a command line tool to quickly search / filter through the openvas scan results.

As an added bonus, you can output a .csv file from an nmap scan using gnxparse.py and feed it to goxparse.py to provide an inline comparison of open ports.

$ ./goxparse.py --help
usage: goxparse.py filename.xml [OPTIONS]

Glens OpenVas XML Parser (goxparse)

positional arguments:
  file  File containing OpenVAS XML report

optional arguments:
  -h, --help                show this help message and exit
  -i, -ips                  Output unfiltered list of scanned ipv4 addresses
  -host [HOSTIP]            Host to generate a report for
  -cvssmin [CVSSMIN]        Minimum CVSS level to report
  -cvssmax [CVSSMAX]        Maximum CVSS level to report
  -threatlevel [THREAT]     Threat Level to match, LOG/LOW/MEDIUM/HIGH/CRITICAL
  -matchfile [MATCHFILE]        .csv file from which to match open ports, in format HOSTIP,port1,port2,port3
  -v, --version         show program's version number and exit

usage examples:
        goxparse.py ./scan.xml -ips
        goxparse.py ./scan.xml -host <HOSTIP>
        goxparse.py ./scan.xml -cvssmin 5 -cvssmax 8
        goxparse.py ./scan.xml -threatlevel HIGH

 

You can get goxparse from the bitbucket repo here.

infosec python technical

Parsing and Merging Nmap XML Report Files in Python

Here are a couple of tools I wrote in python to parse and merge/ join Nmap .xml report files.

TL;DR:

  • gnxparse.py outputs discovered host, port info from nmap .xml, optionally in the form of nmap command(s) to re-scan hosts.
  • gnxmerge.py glues multiple (<host> sections from) nmap XML reports together.
  • You can download them from the git repo here.

Problem:

Nmap is great for network auditing. Scanning from an internal, privileged, and/or fast network location (eg inside your firewall) is straightforward and fast, but doesn’t give you the whole picture – which discovered hosts and services are exposed from a different – eg external/public network.

To get this info, you could do a firewall config audit, but if you don’t have this access or just want to do a functional test of the firewall, you need to run another scan. For the same accuracy, you’ll want the full range (1-65535) portscan, and this takes time. Also, this kind of scan is noisy and may generate a lot of firewall/ips logs. Lastly, if you traverse an IPS/IDS with such a noisy scan, it may drop you as malicious, and the rest of the results are lost.

Solution:

An alternative approach is to do a full scan internally, and use the results to make a much quieter external scan targeting only known live hosts and services. gnxparse.py can generate nmap ‘rescan’ commands to run from outside the firewall, and gnxmerge.py helps tidy up the results by merging the multiple output files back into a single report.

The workflow goes something like this:

  1. Perform a thorough scan of your publically routable subnets from a location inside the firewall. (Using some fairly well tuned host discovery options in nmap, it takes me about 3 hours to scan the full port range on ~1000 hosts on a fast internal network.)
  2. Run gnxparse.py with the ‘rescan’ option on the .xml file generated from the internal nmap scan. This will output a bash script with individual nmap commands to probe only those hosts and services found to be up.
  3. Copy the script to an external host with nmap and run it. (For all discovered services on the ~1000 hosts scaned earlier, it takes me only about five minutes to do a re-scan).
  4. Run gnxmerge.py on the folder of individual .xml reports generated (one per host) if you need to produce a single Nmap XML report file for any reason, for example to load up in Zenmap[1] and review which of your services are exposed externally.

Hopefully someone else finds these tools useful. You can download  gnxparse.py and gnxmerge.py from my GNXTools repo on bitbucket.

 

[1] I am aware Zenmap can also load multiple nmap report files for viewing; though it does not merge/save.