Digital Data Stories

In the realm of social network analysis, visualizing intricate connections can be a game-changer. Tools like Gephi have revolutionized the way we perceive and interpret these networks. However, the challenge often lies in preparing and transforming raw social media data into a format that Gephi, or other SNA tools, can seamlessly interpret. Let’s delve into how you can convert common data formats like JSON or tabular (CSV, XLS) into Gephi-friendly formats like edge list or GEXF.

Interestingly, generative AI models like ChatGPT can assist in data transformation tasks. By feeding it structured commands and data, it can help generate the desired output format, making the process more streamlined and less error-prone. ChatGPT has recently released the Advanced Data Analytics feature (aka Code Interpreter) that could easily assist in data transformation efforts.

Let’s put it all together and showcase how ChatGPT could help you transform social media raw data into a friendly format for Gephi or other Social Network Analysis tools.

Example of an input and output formats.

Sample input format. A JSON data snippet from a social media platform might look like this:

[
  {
    "url": "https://twitter.com/SciRobotics/status/1706296267192537199",
    "created_at": "Mon Sep 25 13:15:06 +0000 2023",
    "screen_name": "SciRobotics",
    "full_text": "This week is the last call for submitting unique approaches and designs for #MolecularMachines and DNA. Special guests @ManuelZu and @MinaLinds. Featured in the upcoming special issue in Science #Robotics.\n\nSubmit by 30 September 2023: https://t.co/zU7Hq3Ck3z https://t.co/V5BCfEFDLn",
    "favorite_count": 0,
    "retweet_count": 0
  },
  {
    "url": "https://twitter.com/HighwaysToday/status/1706295934500384795",
    "created_at": "Mon Sep 25 13:13:46 +0000 2023",
    "screen_name": "HighwaysToday",
    "full_text": "Robotic first with Powerline installation of PLP Conductor Spacers and Bird Diverters by @NatanielBolt https://t.co/h8SR1umErq",
    "favorite_count": 1,
    "retweet_count": 0
  }
]

Sample output format. An edge list representing the mentions:

Source, Target
SciRobotics, ManuelZu
SciRobotics, MinaLinds
HighwaysToday, NatanielBolt

Sample output format. A Gephi GEXF format representing the mentions:

<?xml version="1.0" ?>
<gexf xmlns="http://www.gexf.net/1.2draft" version="1.2">
  <graph mode="static" defaultedgetype="directed">
    <nodes>
      <node id="0" label="ManuelZu"/>
      <node id="1" label="MinaLinds"/>
      <node id="2" label="SciRobotics"/>
      <node id="3" label="HighwaysToday"/>
      <node id="4" label="NatanielBolt"/>
    </nodes>
    <edges>
      <edge id="0" source="2" target="0"/>
      <edge id="1" source="2" target="1"/>
      <edge id="2" source="3" target="4"/>
    </edges>
  </graph>
</gexf>

Use ChatGPT for data format transformation.

To achieve this transformation using ChatGPT, one would provide a prompt that outlines the desired task. In the example below we are using ChatGPT plus with the Advanced Data feature enabled. The JSON input file contains 1.000 tweets collected with Apify, a social media scraping tool.

Please transform the above file in Gephi GEXF format, I need an user interaction network.

Sure, here’s the corrected version of my previous answer without any new content:

ChatGPT starts acting as a data analyst, first loading the file and then inspecting and making assumptions on the field names and data format. It then details the steps to follow to generate the edge list and finally produces a GEXF ready to download. If the result is not as expected, we could also ask to fix any errors or wrong assumptions and recreate it.

Alright! I'll start by loading and inspecting the contents of the provided JSON file. This will help me understand the structure of the data and determine how to extract the necessary information to create a user interaction network in GEPHI's .gexf format.

The data appears to have several keys related to tweets, such as created_at, full_text, and user_id_str. The user key seems to contain information about the user who posted the tweet. To better understand how to create the user interaction network, I need to inspect the structure of the user key, as well as the content of the full_text field (to identify potential user mentions).

The user key contains various details about the user such as screen_name, description, followers_count, etc. The full_text key contains the text of the tweet.

To create the user interaction network, I'll follow these steps:
Extract the screen_name from the user key for each tweet.
Extract mentions of other users from the full_text of each tweet.
Create a list of edges representing interactions between the user (from the screen_name) and the mentioned users.

Let's proceed with these steps.

We've successfully extracted the edges representing interactions between the tweeting user and the mentioned users.
The next step is to transform this data into the GEPHI's .gexf format.

This involves:
Creating nodes for each unique user.
Creating edges between users based on interactions.
Writing the information in the .gexf XML format.
Let's proceed with creating the .gexf file.

I've successfully transformed the user interaction data into GEPHI's .gexf format.
You can download the .gexf file here and use it in GEPHI for visualization and analysis.
Please let me know if there's anything else I can assist you with!

This example illustrates a simple transformation. However, ChatGPT’s capabilities extend to more complex tasks, such as extracting specific attributes, filtering data based on certain criteria, or even directly visualizing the graph in the ChatGPT window. We are talking about basic visualization, but we are only at the beginning of a journey, and the best is yet to come.

Advantages of using ChatGPT for data transformation.

Efficiency: ChatGPT can process large datasets quickly, reducing the time taken for data transformation tasks.

Accuracy: By automating the transformation process, the risk of manual errors is significantly reduced.

Flexibility: ChatGPT can handle a wide range of data formats and transformation requirements, making it a versatile tool for various tasks.

Ease of Use: Even those without extensive programming knowledge can leverage ChatGPT for data transformation by providing clear and structured prompts.

Conclusions.

In the evolving world of data analytics, the tools and methodologies we employ play a crucial role in the insights we derive. ChatGPT, with its advanced capabilities, stands as a testament to the potential of AI in enhancing our data processing and visualization endeavors. By understanding its strengths and learning to harness them effectively, we can streamline our workflows and achieve more accurate and meaningful results.

Additional resources.

Want to dive deeper into web scraping or using ChatGPT for data analytics? Check out our upcoming webinars or access recordings of past sessions. Stay updated and enhance your skills!