WhoisXML API Blog

WhoisXML API has released a Python script that tells you whether a domain’s top-level domain (TLD) (including multi-level TLDs like .co.uk) supports RDAP, WHOIS, both protocols, or neither, based on a reference TLD dataset.

The script is useful for anyone performing programmatic WHOIS lookups who wants to know in advance which protocol a TLD responds to. It works in two modes and requires a TLD reference CSV (which you can get from rdap.wxapros.com) as its first input:

Ed Gibbs, VP of Research, whoisxmlapi.com

Issue Overview

CSV files containing CR/LF (carriage return/line feed) line endings can cause parsing errors with certain CSV-to-JSON conversion tools. This commonly occurs when CSV files are created on Windows systems, which use CR/LF (\r\n) line endings, while many Unix-based tools expect only LF (\n) line endings.  This applies to various datafeeds WHOISXMLAPI provides such as WHOIS, Netblocks, and IP Geolocation.

Introduction

Overview

DynamoDB is Amazon’s fully managed NoSQL database service that’s designed for high-performance applications at scale. DynamoDB is known for its single-digit millisecond response times, automatic scaling, and built-in security features.

WhoisXML API has recently created a Python script to help Reverse NS API users capture the entire query output without being restricted to 300 results through automatic pagination.

The script essentially removes the limitation of standard API pagination, automatically retrieving all subsequent pages of the query output instead of the user manually requesting each page. This capability enables users to fully automate processes and pipelines that rely on Reverse NS API data.

WhoisXML API recently created a Python script to help users of First Watch Malicious Domains Data Feed, also known as “First Watch,” transform predictive domain intelligence into a machine-readable format—STIX 2.1—for automated data processing.

The script reads domain names from First Watch files, converts them to STIX 2.1 Indicator objects with domain-name observable types, and generates a TAXII 2.1-compatible STIX bundle. The output is downloadable as a JSON file in TAXII format.

Abstract

This document outlines the setup of a PostgreSQL database on Ubuntu Linux to efficiently manage and query WHOISXMLAPI’s Premium DNS database. Designed to store and analyze billions of DNS records, this database will handle large-scale data ingestion, facilitate rapid data retrieval, and support extensive analytical operations. PostgreSQL’s robust performance, scalability, and support for advanced indexing make it ideal for managing DNS data, while its compatibility with open-source tools provides a flexible environment for future scaling and data processing.

Abstract

Apache Cassandra is a highly scalable, distributed NoSQL database designed for handling massive volumes of data across many commodity servers without a single point of failure. Its decentralized nature and robust architecture make it particularly well-suited for applications that require high availability, fault tolerance, and horizontal scalability. Cassandra is engineered to handle very large datasets, supporting billions of records with ease, making it an ideal choice for organizations dealing with large-scale, real-time applications such as time-series data, IoT data, and customer logs. Through its use of a partitioned architecture and the ability to add nodes seamlessly as data grows, Cassandra offers an efficient means of managing big data with low latency and high throughput.

Abstract

This white paper provides a comprehensive guide for importing First Watch Malicious Domain data from a CSV file into a MySQL database and indexing it on the `domainName` column. The steps outlined here cover the creation of a database, the table structure design, and data import using MySQL. By following this approach, users can efficiently handle domain-related datasets for querying and analysis purposes.