Data is the foundation of most programming projects, regardless of discipline or purpose. In this chapter, we will explore what data is, the difference between data and information, the types of data, and how we work with data in our disciplines and Python. When we engage with data in Python, we can think of data as units of information. The words data and information are used interchangeably, but they are not the same. Here are general definitions of the two:

With these definitions in mind, we will learn features of Python that support the creation, collection, and storage of data. We use these features in preparation for the next step of the Input ⇨ Processing ⇨ Output cycle of processing.

There are many different ways to create and collect data. We will explore few examples of using Python to create or collect data in a Python program.

• Hardcoding data values directly in our Python code. While this is not a common practice in production systems, we often hardcode values in our code to test our code and algorithms during development.
• User Prompts are used in interactive Python programs. Prompts request data from the user by printing a prompt on the screen and waiting until the user has entered a response. When the user presses Enter, we can use their entry in our Python program.
• Data files contain from one to many rows of data that we can read into our Python program for processing.
• Databases are storage systems for large quantities of data that we can access using Python to request data based on specified criteria (called a query).
• Web Scraping is an approach for collecting data by scrapping (extracting) data from websites programmatically using Python.
• APIs (Application Programming Interfaces) are specialized tools for communication between two computer programs. This facilitates our Python programs' ability to communicate with APIs that are made available by various systems and web platforms (like Google Maps, the National Weather Services, and others).
• Sensors & hardware devices often include the ability to transmit data to external programs, which allows us to access that data from our Python programs.
• Online surveys & forms generate data when users complete these data collection tools online and submit their responses. We can use Python to process those responses to produce various storage and output tasks.
• Computer, network, and web monitoring tools collect data used to manage systems. Python programs process that data in order to present information to system admininstrators.

Example of Hardcoded Data

As indicated above, hardcoded data within code is most often considered a bad programming practice. However, when you're learning to program and during development we often use hardcoded data values with the understanding that when we create live production-level programs we minimize or eliminate the use of hardcoding. Figure 1 depicts an Input ⇨ Processing ⇨ Output scenario in which we hardcode a data value into a Python program, which is processed by our program, and then an output is printed to the screen. In this example, we hardcode a data value (10), which will be used as our input. The process performs a calculation (black boxed in this image) that produces an output, in this case, a statement that reports the area of a circle calculated from the hardcoded data value of radius.

Example of Data Entered by a User

In contrast to hardcoded data, prompting a user for a data value makes our programs more interactive and flexible. Figure 2 depicts the same scenario as seen in Figure 1, a program that calculates the area of a circle. In this example, though, the radius data value comes from a user entering a value. We use the Python input() function to prompt the user for values. The process performs a calculation (black boxed in this image) that produces an output, in this case, a statement that reports the area of a circle calculated based on the radius data value entered by the user.

Example of Data Read from a Data File

In this third example, we'll consider a different scenario. Another way we collect data is by reading a data file. Figure 3 depicts a data file containing a summary list of student academic majors in a class. Each line of the data file includes a major and a number, the number of students in the class who have declared that major. The black box process in our Python program produces a chart visually displaying each major and the number of students in each as a horizontal bar chart.

In addition to understanding the sources of data, we also need to be aware of different types of data for proper data collection, analysis, and interpretation in various fields, ranging from research to programming to business intelligence.

• Quantitative data
  • Description: Quantitative data is numerical, allowing for measurement and comparison. It is typically used in statistical analysis and can be further classified into discrete and continuous data.
  • Examples:
    • Discrete Data: Counts of items, such as the number of students in a class (e.g., 30).
    • Continuous Data: Measurements that can take any value within a range, like the height of students in meters.
• Qualitative Data
  • Description: This type of data is descriptive and characterizes attributes or properties that are not numerical. It can be observed but not measured.
  • Examples:
    • Nominal Data: Data without a natural order or ranking, such as types of cuisine (Italian, Chinese, Mexican).
    • Ordinal Data: Data with a set order or scale, but without a standard interval, like survey responses (poor, fair, good, very good, excellent).
• Primary Data
  • Description: Primary data is collected firsthand by the researcher for a specific purpose. It is original and collected at the source.
  • Examples:
    • Survey responses collected by a marketer to assess customer satisfaction with a new product.
    • Laboratory experiment results in a scientific study.
• Secondary Data
  • Description: This data is not collected directly by the user but obtained from existing sources. It was collected for other purpose but is being used for a different analysis.
  • Examples:
    • Census data used by a business for market analysis.
    • Historical sales data used for trend analysis.
• Structured Data
  • Description: Structured data is highly organized and formatted in a way that is easily searchable in databases. It adheres to a specific format or schema.
  • Examples:
    • Database tables with rows and columns, like a student enrollment database.
    • Spreadsheets with defined data types for each column.
• Unstructured Data
  • Description: This type of data lacks a predefined format or structure, making it more complex to analyze and process.
  • Examples:
    • Text files, such as emails or product reviews.
    • Multimedia content like images, audio, and video files.
• Semi-structured Data
  • Description: Semi-structured data does not reside in a relational database but has some organizational properties that make it easier to analyze than unstructured data.
  • Examples:
    • JSON and XML files.
    • Emails that contain both structured elements (like sender, recipient, date) and unstructured text bodies.
• Time-Series Data
  • Description: This data is a sequence of data points collected at consistently over time. It is used to analyze trends, patterns, and future forecasting.
  • Examples:
    • Stock market prices recorded throughout the trading day.
    • Daily temperature readings.
• Cross-sectional Data
  • Description: Data collected from multiple sources at a single point in time. It is used to analyze and compare different variables at a specific moment.
  • Examples:
    • A survey of consumer preferences taken at a particular date.
    • Data on the GDP of different countries for a given year.
• Big Data
  • Description: Refers to extremely large datasets that traditional data processing software cannot handle efficiently. Big data is characterized by its volume, velocity, and variety.
  • Examples:
    • Social media data with millions of posts, likes, and comments.
    • Sensor data from Internet of Things (IoT) devices.

This page briefly introduced data, gave examples of how data is created or collected, and provided a list of different types of data we might encounter. Next, we will learn about each type of data we can work with and how to manage and manipulate that data in Python.