WhoisXML API Blog

Managing a Passive DNS Database Using PostgreSQL

Abstract

This document outlines the setup of a PostgreSQL database on Ubuntu Linux to efficiently manage and query WHOISXMLAPI’s Premium DNS database. Designed to store and analyze billions of DNS records, this database will handle large-scale data ingestion, facilitate rapid data retrieval, and support extensive analytical operations. PostgreSQL's robust performance, scalability, and support for advanced indexing make it ideal for managing DNS data, while its compatibility with open-source tools provides a flexible environment for future scaling and data processing.

Managing a Passive DNS Database Using Apache Cassandra

Abstract

Apache Cassandra is a highly scalable, distributed NoSQL database designed for handling massive volumes of data across many commodity servers without a single point of failure. Its decentralized nature and robust architecture make it particularly well-suited for applications that require high availability, fault tolerance, and horizontal scalability. Cassandra is engineered to handle very large datasets, supporting billions of records with ease, making it an ideal choice for organizations dealing with large-scale, real-time applications such as time-series data, IoT data, and customer logs. Through its use of a partitioned architecture and the ability to add nodes seamlessly as data grows, Cassandra offers an efficient means of managing big data with low latency and high throughput.

October 2024: Domain Activity Highlights

The WhoisXML API research team analyzed more than 8.2 million domains registered between 1 and 31 October 2024 to identify the most popular registrars, top-level domain (TLD) extensions, and other global domain registration trends.

We also determined the top TLD extensions used by the more than 57.4 billion domains from our DNS database’s A record full file released in the same month.

Next, we studied the top TLDs of more than 1.0 million domains detected as indicators of compromise (IoCs) in October.

Finally, we summed up our findings and provided links to the threat reports produced using DNS, IP, and domain intelligence sources during the period.

DNS Database Download Is Now Reinforced with Wildcard and Active Fields

We are excited to announce that the Standard and Premium DNS Database files from DNS Database Download are now enriched with two new columns, namely, wildcard and active. These additions allow you to determine if a DNS record is part of a wildcard entry and check if a domain name or subdomain is active based on its most recent resolution status.

Quarterly WHOIS Database Download Files Are Now Available on Snowflake

We are excited to announce that Snowflake users can now request access to WHOIS Database Download’s quarterly gTLD and ccTLD files on the platform, available in three formats:

• Simple: The simple CSV file contains the domain name, registrar name and email address, WHOIS and name servers, creation and expiration dates, and registrant and administrative contact details.

• Regular: The regular CSV file contains all the fields in the Simple file, along with information about the billing, technical, and zone contacts.

• Full: This file format contains all fields in the Simple file, in addition to the raw text from the WHOIS registry and registrar.

Importing and Indexing First Watch Malicious Domains Data Feed into MySQL

Abstract

This white paper provides a comprehensive guide for importing First Watch Malicious Domain data from a CSV file into a MySQL database and indexing it on the `domainName` column. The steps outlined here cover the creation of a database, the table structure design, and data import using MySQL. By following this approach, users can efficiently handle domain-related datasets for querying and analysis purposes.

WhoisXML API Launches New Passive DNS Product DNS Chronicle API

We are excited to introduce DNS Chronicle API, the latest addition to our passive DNS offerings. This API release enables use cases such as proactive threat detection and attack surface discovery by providing visibility into the complete DNS history of a domain or an IP address.

With DNS Chronicle API, users can perform two types of passive DNS queries, namely:

• Forward search: Using any FQDN as a search string, users can retrieve its historical A and AAAA records.

• Reverse search: Users can obtain a list of all the FQDNs associated with a given IP address.

September 2024: Domain Activity Highlights

The WhoisXML API research team analyzed more than 7.1 million domains registered between 1 and 30 September 2024 to identify the most popular registrars, top-level domain (TLD) extensions, and other global domain registration trends.

We also determined the top TLD extensions used by the more than 60.1 billion domains from our DNS database’s A record full file released in the same month.

Next, we studied the top TLDs and associated threat types of more than 1.0 million domains detected as indicators of compromise (IoCs) in September.

Finally, we summed up our findings and provided links to the threat reports produced using DNS, IP, and domain intelligence sources during the period.