Introduction and Text Data Collection
Welcome to today’s lesson! As data science and machine learning professionals, particularly in the Natural Language Processing (NLP) field, we often deal with textual data. Today, we dive into the ‘Introduction to Textual Data Collection’. Specifically, we’ll explore how to collect, understand and analyze text data using Python.
Textual data is usually unstructured, being much harder to analyze than structured data. It can take many forms, such as emails, social media posts, books, or transcripts of conversations. Understanding how to handle such data is a critical part of building effective machine learning models, especially for text classification tasks where we ‘classify’ or categorize texts. The quality of the data we use for these tasks is of utmost importance. Better, well-structured data leads to models that perform better.
The 20 Newsgroups Dataset
The dataset we’ll be working with in today’s lesson is the 20 Newsgroups dataset. For some historical background, newsgroups were the precursors to modern internet forums, where people gathered to discuss specific topics. In our case, the dataset consists of approximately 20,000 documents from newsgroup discussions. These texts were originally exchanged through Usenet, a global discussion system that predates many modern Internet forums.
The dataset is divided nearly evenly across 20 different newsgroups, each corresponding to a separate topic - this segmentation is one of the main reasons why it is especially useful for text classification tasks. The separation of data makes it excellent for training models to distinguish between different classes, or in our case, newsgroup topics.
From science and religion to politics and sports, the topics covered provide a diversified range of discussions. This diversity adds another layer of complexity and richness, similar to what we might experience with real-world data.
Fetching and Understanding the Data Structure
To load this dataset, we use the fetch_20newsgroups() function from the sklearn.datasets module in Python. This function retrieves the 20 newsgroup dataset in a format that’s useful for machine learning purposes. Let’s fetch and examine the dataset.
First, let’s import the necessary libraries and fetch the data:
| |
The datasets fetched from sklearn typically have three attributes—data, target, and target_names. data refers to the actual content, target refers to the labels for the texts, and target_names provides names for the target labels.
Next, let’s understand the structure of the fetched data:
Python
CopyPlay
| |
We are fetching the data and observing the type of the data and target. The type of data tells us what kind of data structure is used to store the text data while the type of target shouts what type of structure is used to store the labels. Here is what the output looks like:
| |
As printed out, the data is stored as a list, and target as a numpy array.
Diving Into Data Exploration
Now, let’s explore the data points, target variables and the potential classes in the dataset:
| |
We get the length of the data list to fetch the number of data points. Also, we get the length of the target array. Lastly, we fetch the possible classes or newsgroups in the dataset. Here is what we get:
| |
Sample Data Preview
Lastly, let’s fetch and understand what a sample data point and its corresponding label looks like:
| |
The Article fetched is the 10th article in the dataset and Corresponding Topic is the actual topic that the article belongs to. Here’s the output:
| |
Lesson Summary
Nice work! Through today’s lesson, you’ve learned to fetch and analyze text data for text classification. If you’ve followed along, you should now understand the structure of text data and how to fetch and analyze it using Python.
But our journey to text classification is just starting. In upcoming lessons, we’ll dive deeper into related topics such as cleaning textual data, handling missing values, and restructuring textual data for analysis. Each step forward improves your expertise in text classification. Keep going!
「Practice1」Explore More of the 20 Newsgroups Dataset
「Practice2」Uncover the End of 20 Newsgroups Dataset
Celestial Traveler, your journey continues! Fill in the blanks (____) to import and explore our dataset. We aim to extract and display the last three articles and their corresponding topics. Can you reveal what’s at the end of our dataset?
| |
“Here is the completed code to import and explore the dataset, extracting and displaying the last three articles and their corresponding topics.”
| |
「Practice3」Fetch Specific Categories from Dataset
Celestial Traveler, let’s narrow down our data collection. Modify the provided code to fetch only the 'alt.atheism' and 'talk.religion.misc' categories from our dataset. Then, display the first two articles from these categories along with their corresponding labels.
天体旅行者，让我们缩小数据收集范围。修改提供的代码，使其仅从我们的数据集中获取 'alt.atheism' 和 'talk.religion.misc' 类别。然后，显示来自这些类别的前两篇文章
| |
“Here is the modified code to fetch only the 'alt.atheism' and 'talk.religion.misc' categories from the dataset, and to display the first two articles along with their corresponding labels.”
| |
输出
| |
「Practice」Fetching the Third Article from Dataset
Well done, Stellar Navigator! Next, fill in the missing line in the code below to fetch and display the third article from the 20 Newsgroups dataset with its corresponding topic. Prepare your spacecraft for another adventure in data exploration!
干得好，星际导航员！接下来，填写以下代码中缺少的行，以获取并显示 20 Newsgroups 数据集中第三篇文章及其对应主题。准备好你的宇宙飞船，开始另一场数据探索冒险之旅吧！
| |
“Here is the completed code to fetch and display the third article from the 20 Newsgroups dataset along with its corresponding topic.”
| |
输出
| |
「Practice」Exploring Text Length in Newsgroups Dataset
Great job, Space Voyager! Now, as a final task, write a Python script that calculates and displays the lengths of the first five articles (in terms of the number of characters) from the 20 Newsgroups dataset.
| |
“Here is the completed Python script to calculate and display the lengths of the first five articles in terms of the number of characters from the 20 Newsgroups dataset.”
| |
Lesson 2：Mastering Text Cleaning for NLP: Techniques and Applications
Introduction 引言
Welcome to today’s lesson on Text Cleaning Techniques! In any Natural Language Processing (NLP) project, the quality of your results depends heavily on the quality of your input. Hence, cleaning our textual data becomes critical for the accuracy of our project. Our main objective for today is to delve into how to clean textual data using Python. By the end of this session, you will be comfortable with creating and applying a simple text cleaning pipeline in Python.
欢迎来到今天关于文本清理技术的课程！在任何自然语言处理 (NLP) 项目中，结果的质量在很大程度上取决于输入的质量。因此，清理文本数据对于项目的准确性至关重要。我们今天的主要目标是深入研究如何使用 Python 清理文本数据。在本课程结束时，您将能够轻松地使用 Python 创建和应用简单的文本清理管道。
Understanding Text Cleaning
理解文本清洗
Text cleaning is essential in NLP, involving the preparation of text data for analysis. Why is it necessary? Imagine trying to perform text classification on social media posts; people often use colloquial language, abbreviations, and emojis. In many cases, posts might also be in different languages. These variations make it challenging for machines to understand context without undergoing preprocessing.
文本清洗在自然语言处理 (NLP) 中至关重要，涉及为分析准备文本数据。为什么需要它？想象一下尝试对社交媒体帖子进行文本分类；人们经常使用口语、缩写和表情符号。在许多情况下，帖子也可能使用不同的语言。这些差异使得机器在未经预处理的情况下难以理解上下文。
We get rid of superfluous variations and distractions to make the text understandable for algorithms, thereby increasing accuracy. These distractions could range from punctuation, special symbols, numbers, to even common words that do not carry significant meaning (commonly referred to as “stop words”).
我们去除多余的变化和干扰因素，使文本易于算法理解，从而提高准确性。这些干扰因素包括标点符号、特殊符号、数字，甚至是不具有重要意义的常见词（通常称为“停用词”）。
Python’s Regex (Regular Expression) library, re, is an ideal tool for such text cleaning tasks, as it is specifically designed to work with string patterns. Within this library, we will be using re.sub, a method employed to replace parts of a string. This method operates by accepting three arguments: re.sub(pattern, repl, string). Here, pattern is the character pattern we’re looking to replace, repl is the replacement string, and string is the text being processed. In essence, any part of the string argument that matches the pattern argument gets replaced by the repl argument.
Python 的正则表达式 (Regex) 库 re 是此类文本清理任务的理想工具，因为它专门用于处理字符串模式。在这个库中，我们将使用 re.sub 方法来替换字符串的某些部分。此方法接受三个参数： re.sub(pattern, repl, string) 。其中， pattern 是我们要替换的字符模式， repl 是替换字符串， string 是正在处理的文本。本质上， string 参数中与 pattern 参数匹配的任何部分都将被 repl 参数替换。
As we proceed, a clearer understanding of the functionality and application of re.sub will be provided. Now, let’s delve into it!
随着我们的深入，我们将对 re.sub 的功能和应用有更清晰的了解。现在，让我们开始吧！
Text Cleaning Process 文本清理流程
The text cleaning process comprises multiple steps where each step aims to reduce the complexity of the text. Let’s take you through the process using a Python function, clean_text.
文本清理过程包含多个步骤，每个步骤都旨在降低文本的复杂性。让我们通过一个 Python 函数 clean_text 来带您了解整个过程。
| |
In the function above we can see how each line corresponds to a step in the cleaning process:
在上面的函数中，我们可以看到每一行是如何对应于清洁过程中的一个步骤的：
- Lowercase: We convert all text to lower case, so every word looks the same unless it carries a different meaning. This way, words like ‘The’ and ’the’ are no longer seen as different.
小写：我们将所有文本转换为小写，因此每个单词看起来都一样，除非它具有不同的含义。这样，“The”和“the”就不再被视为不同的词。 - Email addresses: Email addresses don’t usually provide useful information unless we’re specifically looking for them. This line of code removes any email addresses found.
电子邮件地址：电子邮件地址通常不会提供有用信息，除非我们专门查找它们。这行代码会删除找到的任何电子邮件地址。 - URLs: Similarly, URLs are removed as they are typically not useful in text classification tasks.
URL：类似地，URL 通常在文本分类任务中没有用处，因此会被删除。 - Special Characters: We remove any non-word characters (
\W) and replace it with space using regex. This includes special characters and punctuation.
特殊字符：我们使用正则表达式删除任何非单词字符（\W）并将其替换为空格。这包括特殊字符和标点符号。 - Numbers: We’re dealing with text data, so numbers are also considered distractions unless they carry significant meaning.
数字：我们处理的是文本数据，因此除非数字具有重要意义，否则它们也被视为干扰因素。 - Extra spaces: Any resulting extra spaces from the previous steps are removed.
从先前步骤产生的任何额外空格都将被删除。
Let’s go ahead and run this function on some demo input to see it in action!
让我们继续，在一些演示输入上运行此函数，看看它的实际效果！
| |
The output of the above code will be:
以上代码的输出将是：
| |
Implementing Text Cleaning Function
实现文本清洗功能
Now that you are familiar with the workings of the function let’s implement it in the 20 Newsgroups dataset.
现在你已经熟悉了函数的工作原理，让我们在 20 Newsgroups 数据集中实现它。
To apply our cleaning function on the dataset, we will make use of the DataFrame data structure from Pandas, another powerful data manipulation tool in Python.
为了在数据集上应用我们的清洗函数，我们将利用 Pandas 中的 DataFrame 数据结构，它是 Python 中另一个强大的数据操作工具。
| |
The output of the above code will be:
以上代码的输出将是：
| |
In this code, we’re applying the clean_text function to each ’text’ in our DataFrame using the apply function. The apply function passes every value of the DataFrame column to the clean_text function one by one.
在这段代码中，我们使用 apply 函数将 clean_text 函数应用于 DataFrame 中的每个“文本”。 apply 函数将 DataFrame 列的每个值逐个传递给 clean_text 函数。
Understanding Effectiveness of Text Cleaning Function
理解文本清洗功能的有效性
We want to understand the impact of our text cleaning function. We can achieve this by looking at our text before and after cleaning. Let’s use some new examples:
我们想要理解文本清洗函数的影响。我们可以通过查看清洗前后的文本内容来实现这一点。让我们使用一些新的例子：
| |
The output of the above code will be:
以上代码的输出将是：
| |
In the example above, you will see that our function successfully transforms all text to lower case and removes punctuation, digits, and email addresses!
在上面的例子中，你会看到我们的函数成功地将所有文本转换为小写，并删除了标点符号、数字和电子邮件地址！
Lesson Summary and Practice Exercises
课文总结和练习题
Today we delved into the text cleaning process in Natural Language Processing. We shared why it is necessary and how to implement it in Python. We then applied our text cleaning function on a textual dataset.
今天我们深入探讨了自然语言处理中的文本清洗过程。我们分享了为什么需要文本清洗以及如何在 Python 中实现它。然后，我们将文本清洗函数应用于一个文本数据集。
We have a few exercises lined up based on what we learned today. Keep swimming ahead, and remember, you learn the most by doing. Happy cleaning!
我们准备了一些练习，都是基于今天所学的内容。继续加油练习，记住，实践出真知。祝你顺利完成！
「Practice1」
Well done, Space Voyager! Now, to further explore the workings of our text cleaning function, let’s use a different sentence. Replace the first sentence in the test_texts list with the phrase “I love learning at CodeSignal; it’s so interactive and fun!”. Then run the clean_text function with the updated list.
干得好，太空旅行者！现在，为了进一步探索我们文本清理功能的工作原理，让我们使用不同的句子。将 test_texts 列表中的第一句话替换为“我喜欢在 Co 学习
干得好，太空旅行家！现在，为了进一步探索文本清理函数的工作原理，让我们使用不同的句子。将 test_texts 列表中的第一句话替换为“我喜欢在代码学习
| |
输出
| |
「Practice2」Filling in Python Functions and Regex Patterns
Superb job! Now, let’s ensure you have grasped the core Python functions and regex patterns used in text cleaning.
干得好！现在，让我们来确保你已经掌握了文本清理中使用的核心 Python 函数和正则表达式。
Fill in the blanks (____) to transform all text to lowercase and remove extra spaces.
| |
解释
| |
输出
| |
「Practice2」Filling in Python Functions and Regex Patterns
Superb job! Now, let’s ensure you have grasped the core Python functions and regex patterns used in text cleaning.
Fill in the blanks (____) to transform all text to lowercase and remove extra spaces.
| |
解释 Certainly! Here’s the completed Python code with the blanks filled:
| |
Explanation:
- The
lower()method converts all text to lowercase. - The regex pattern
r'\s+'matches one or more whitespace characters, which are then replaced by a single space to remove extra spaces.
输出
| |
Practice 3：Mastering Text Cleaning with Python Regex
Charming maneuvering, Stellar Navigator!
Now, I want to see more of your Python prowess. Replace the missing parts (____), ensuring that they perform the right operations. We aim to transform all text to lowercase and eliminate email addresses, URLs, special characters, digits, and extra spaces.
| |
解释 To clean the text by transforming it to lowercase and removing email addresses, URLs, special characters, digits, and extra spaces, you can fill in the missing parts of the provided Python code as follows:
| |
Explanation of the substitutions:
text.lower(): Converts the text to lowercase.re.sub(r'\b[\w\.-]+?@\w+?\.\w+?\b', '', text): Removes email addresses by matching common email patterns.re.sub(r'http\S+|www\.\S+', '', text): Removes URLs by matching patterns that start with “http” or “www”.re.sub(r'[^\w\s]', ' ', text): Removes special characters by replacing anything that is not a word character or whitespace with a space.re.sub(r'\d+', '', text): Removes digits by matching one or more digits.re.sub(r'\s+', ' ', text): Replaces multiple spaces with a single space.
This code will transform the given text according to the specified requirements.
「Practice」
Off to a flying start, Celestial Traveler! Your new task requires you to insert missing code into two sections. First, write code that transforms the collected dataset into a DataFrame. Then, ensure that you apply the clean_text function to the DataFrame to clean up the textual data. Bon Voyage!
| |
解释
Certainly! Let’s expand on your query to ensure comprehensive coverage. You are asking for the code needed to transform a dataset collected from the fetch_20newsgroups function into a pandas DataFrame. The task also involves applying a clean_text function to the text data within this DataFrame to clean it up by removing email addresses, URLs, punctuation, digits, and normalizing whitespace. Below is the complete code to achieve this:
- Transform the fetched dataset into a
DataFrame. - Apply the
clean_textfunction to the text column in thisDataFrame.
Here’s the completed code:
| |
Explanation:
- Fetching Data: The
fetch_20newsgroupsfunction fromsklearn.datasetsis used to fetch the dataset. - Creating DataFrame: A pandas
DataFrameis created from the dataset, with a single column namedtext. - Cleaning Text: The
clean_textfunction is applied to each entry in thetextcolumn of the DataFrame to clean the text. - Display Cleaned Data: The first few rows of the cleaned DataFrame are printed to verify the cleaning process.
This code ensures that the dataset is not only structured properly but also cleaned efficiently, ready for any further text processing or analysis tasks. 输出
| |
「Practice」
Excellent work so far, Stellar Navigator! This time, you’ll master how to clean textual data. We aim to transform pieces of text into cleaner versions, making them more suitable for further analysis. Let’s put the focus on the cleaning functions you’ve built previously. Implement a combined function that runs all these cleaning operations in sequence: converting text to lowercase letters, removing email addresses, URLs, special characters, digits, and extra spaces. Finally,apply this unified cleaning function to a dataset.
| |
「Practice 5」：Mastering Text Cleaning with Python Regex on a Dataset
Excellent work so far, Stellar Navigator! This time, you’ll master how to clean textual data. We aim to transform pieces of text into cleaner versions, making them more suitable for further analysis. Let’s put the focus on the cleaning functions you’ve built previously. Implement a combined function that runs all these cleaning operations in sequence: converting text to lowercase letters, removing email addresses, URLs, special characters, digits, and extra spaces. Finally,apply this unified cleaning function to a dataset.
| |
Certainly! Let’s rephrase and expand your request to ensure a comprehensive solution.
You want to implement a combined function that performs a series of text cleaning operations, which include converting text to lowercase, removing email addresses, URLs, special characters, digits, and extra spaces. After defining this unified cleaning function, you will apply it to a dataset and inspect the cleaned text.
Here’s the step-by-step process to achieve this:
- Implement a function
clean_textthat sequentially applies all the cleaning operations. - Fetch a dataset (in this case, the 20 Newsgroups dataset).
- Create a DataFrame from the fetched dataset.
- Apply the
clean_textfunction to the text data in the DataFrame. - Inspect the cleaned text.
Let’s start by defining the clean_text function:
| |
This script does the following:
- Defines each individual cleaning function.
- Creates a
clean_textfunction that applies all the cleaning steps in sequence. - Fetches the 20 Newsgroups dataset.
- Creates a DataFrame from the dataset, with the text data in a column named ’text’.
- Applies the
clean_textfunction to each text entry in the DataFrame. - Prints the first few cleaned text entries to verify the results.
By running this code, you will transform the text data into a cleaner version suitable for further analysis, ensuring it is free from unwanted characters and formatted consistently. Certainly! Let’s rephrase and expand your request to ensure a comprehensive solution.
You want to implement a combined function that performs a series of text cleaning operations, which include converting text to lowercase, removing email addresses, URLs, special characters, digits, and extra spaces. After defining this unified cleaning function, you will apply it to a dataset and inspect the cleaned text.
Here’s the step-by-step process to achieve this:
- Implement a function
clean_textthat sequentially applies all the cleaning operations. - Fetch a dataset (in this case, the 20 Newsgroups dataset).
- Create a DataFrame from the fetched dataset.
- Apply the
clean_textfunction to the text data in the DataFrame. - Inspect the cleaned text.
Let’s start by defining the clean_text function:
| |
This script does the following:
- Defines each individual cleaning function.
- Creates a
clean_textfunction that applies all the cleaning steps in sequence. - Fetches the 20 Newsgroups dataset.
- Creates a DataFrame from the dataset, with the text data in a column named ’text’.
- Applies the
clean_textfunction to each text entry in the DataFrame. - Prints the first few cleaned text entries to verify the results.
By running this code, you will transform the text data into a cleaner version suitable for further analysis, ensuring it is free from unwanted characters and formatted consistently.
lesson
Blast through text preprocessing with ease! 🚀 Keep up the great work - you’re doing stellar!
Introduction
Hello and welcome to this lesson on Removing Stop Words and Stemming! In this lesson, we will dive deep into two essential steps to prepare text data for machine learning models: removing stop words and stemming. These techniques will help us improve the efficiency and accuracy of our models. Let’s get started!
Understanding Stop Words
Stop words in Natural Language Processing (NLP) refer to the most common words in a language. Examples include “and”, “the”, “is”, and others that do not provide significant meaning and are often removed to speed up processing without losing crucial information. For this purpose, Python’s Natural Language Tool Kit (NLTK) provides a pre-defined list of stop words. Let’s have a look:
| |
The output of the above code will be:
| |
Here, the stopwords.words('english') function returns a list of English stop words. You might sometimes need to add domain-specific stop words to this list based on the nature of your text data.
Introduction to Stemming
Stemming is a technique that reduces a word to its root form. Although the stemmed word may not always be a real or grammatically correct word in English, it does help to consolidate different forms of the same word to a common base form, reducing the complexity of text data. This simplification leads to quicker computation and potentially better performance when implementing Natural Language Processing (NLP) algorithms, as there are fewer unique words to consider.
For example, the words “run”, “runs”, “running” might all be stemmed to the common root “run”. This helps our algorithm understand that these words are related and they carry a similar semantic meaning.
Let’s illustrate this with Porter Stemmer, a well-known stemming algorithm from the NLTK library:
| |
The output of the above code will be:
| |
The PorterStemmer class comes with the stem method that takes in a word and returns its root form. In this case, “running” is correctly stemmed to its root word “run”. This form of preprocessing, although it may lead to words that are not recognizable, is a standard practice in text preprocessing for NLP tasks.
Stop Words Removal and Stemming in Action
Having understood stop words and stemming, let’s develop a function that removes stop words and applies stemming to a given text. We will tokenize the text (split it into individual words) and apply these transformations word by word.
| |
The output of the above code will be:
| |
The remove_stopwords_and_stem function does the required processing and provides the cleaned-up text.
Stop Words Removal and Stemming on a Dataset
Let’s implement the above concepts on a real-world text dataset – the 20 Newsgroups Dataset.
| |
The output of the above code will be:
| |
This process can take a while for large datasets, but the output will be much cleaner and easier for a machine learning model to work with.
Summary and Conclusion
And that’s a wrap! In today’s lesson, we’ve learned about stop words and stemming as crucial steps in text preprocessing for machine learning models. We’ve used Python’s NLTK library to work with stop words and perform stemming. We have processed some example sentences and a real-world dataset to practice these concepts.
As we proceed to more advanced NLP tasks, pre-processing techniques like removing stop words and stemming would serve as a solid foundation. In the upcoming lessons, we will delve deeper into handling missing text data and learn about reshaping textual data for analysis. Let’s keep going!
Start practice
「practice1」
Excellent work so far, Stellar Navigator! Now it’s time to tweak the implemented text preprocessing method. Replace the use of LancasterStemmer with PorterStemmer. Remember to import PorterStemmer from nltk.stem. After adjusting your code, run it and observe the differences in your processed text.
| |
解释
好的，我们需要将文本预处理方法中的 LancasterStemmer 替换为 PorterStemmer。让我们导入 PorterStemmer 并修改代码以使用新的词干提取器。这里是调整后的代码：
| |
具体解释
导入库和模块
1 2 3from nltk.corpus import stopwords from nltk.stem import PorterStemmer from nltk.tokenize import word_tokenize我们从
nltk.corpus导入stopwords，从nltk.stem导入PorterStemmer，并从nltk.tokenize导入word_tokenize。定义停止词
1stop_words = set(stopwords.words('english'))我们定义了一组停止词，这些是常见的、在分析中通常会被移除的词（如 “is”, “and”, “the” 等）。
创建 PorterStemmer 实例
1stemmer = PorterStemmer()我们创建了一个
PorterStemmer的实例，用于词干提取。定义去除停止词和词干提取的函数
1 2 3 4def remove_stopwords_and_stem(text): tokenized_text = word_tokenize(text) filtered_text = [stemmer.stem(word) for word in tokenized_text if not word in stop_words] return " ".join(filtered_text)这个函数首先将输入文本进行分词，然后移除停止词并进行词干提取，最后将处理后的词重新组合成字符串。
示例文本和处理结果
1 2 3 4 5example_text = "This is an example text to demonstrate the removal of stop words and stemming." processed_text = remove_stopwords_and_stem(example_text) print(f"Original Text: {example_text}") print(f"Processed Text: {processed_text}")我们使用示例文本来展示处理前后的效果。
运行结果
运行代码后，我们会看到处理前后的文本：
- Original Text: This is an example text to demonstrate the removal of stop words and stemming.
- Processed Text: thi exampl text demonstr remov stop word stem
可以观察到，使用 PorterStemmer 后，文本中的词被简化为它们的词干形式，例如 “demonstrate” 被简化为 “demonstr”。与 LancasterStemmer 相比，PorterStemmer 更为常用且更为温和，在实际应用中更为常见。
「Practice」
Well done on completing the first practice, stellar navigator! Now it’s time to put your skills further to the test.
The skeleton code for this task is already written for you. You just need to fill in the blank spots (____) to make the code run without errors.
| |
当然，让我们一步一步地填补代码中的空白，并解释其中的主要知识点。
填补空白
我们需要填补以下空白：
- 从
nltk.corpus导入stopwords。 - 使用
casual_tokenize函数来分词。 - 使用
stopwords.words('english')获取英文停止词。
填补后的代码如下：
| |
主要知识点解释
导入必要模块
1 2 3from nltk.tokenize import casual_tokenize from nltk.corpus import stopwords import string我们从
nltk.tokenize导入casual_tokenize函数用于分词，从nltk.corpus导入stopwords用于获取停止词，并导入string模块来处理标点符号。获取停止词并更新
1 2stop_words = set(stopwords.words('english')) stop_words.update(string.punctuation)这里，我们使用
stopwords.words('english')获取一组英语停止词，并将其转换为集合以便高效查找。然后，我们使用stop_words.update(string.punctuation)将所有标点符号添加到停止词集合中，以确保它们在后续处理中被移除。示例文本
1ex_text = "Here's a sample sentence to remove stop words from. It has generic and specific words."我们定义了一个示例文本，其中包含了需要处理的单词和标点符号。
分词
1ex_words = casual_tokenize(ex_text)使用
casual_tokenize函数将示例文本分词，这个函数特别适用于处理社交媒体文本，因为它能处理缩略词、表情符号等。移除停止词和标点符号
1filtered_words = [word for word in ex_words if not word.lower() in stop_words]这个列表推导式遍历了所有分词后的单词，移除了所有在
stop_words集合中的单词。我们使用word.lower()确保比较时不区分大小写。输出结果
1print("Filtered words:", filtered_words)最后，我们输出处理后的单词列表，这个列表不包含任何停止词或标点符号。
运行结果
假设代码成功运行，输出将是：
| |
在这个结果中，所有的停止词（如 “Here’s”, “a”, “to”, “from”, “it”, “has”）和标点符号都被移除了，只剩下有意义的单词。这种处理在文本分析和自然语言处理（NLP）中非常重要，有助于提高模型的性能和分析的准确性。
「Practice」
Stellar work so far, Space Voyager! Now it’s time to hone your skills in stemming. Fill in the blank spots (____) to make the code functional. Stem the provided words, and print the stemmed versions. Let’s get processing!
| |
解释 To fill in the blanks for this code using the NLTK library, let’s detail the process of stemming words using the Porter Stemmer from NLTK. The code aims to create an instance of the Porter Stemmer class and use it to stem a list of example words.
Here’s the expanded code with the blanks filled in:
| |
Explanation:
- The
from nltk.stem import PorterStemmerline imports the Porter Stemmer class from the NLTK library. - The
example_wordslist contains words that need to be stemmed. - The
stemmer = PorterStemmer()line creates an instance of the Porter Stemmer class. - The list comprehension
[stemmer.stem(word) for word in example_words]stems each word in theexample_wordslist using thestemmethod of the Porter Stemmer instance. - The
printstatement outputs the stemmed words.
「Practice」
Celestial Traveler, let’s put our text preprocessing skills to the test. Can you add the missing line inside the function remove_stopwords_and_stem to tokenize the given text, remove stop words, stem the remaining words, and return the processed text? Give it a shot!
| |
解释
Celestial Traveler, let’s put our text preprocessing skills to the test. Can you add the missing line inside the function remove_stopwords_and_stem to tokenize the given text, remove stop words, stem the remaining words, and return the processed text? Give it a shot!
| |
输出
| |
「Practice」
Good job, Stellar Navigator! Could you now define the stop words and instantiate the PorterStemmer using NLTK’s predefined English stop words and PorterStemmer? Additionally, could you implement the tokenization of the text, remove stop words, apply stemming, and return the processed text in the function remove_stopwords_and_stem? After doing so, you will preprocess the first newsgroup article and display it. The stars are in your favor!
| |
解释
Certainly, let’s complete the clean_text and remove_stopwords_and_stem functions by defining stop words, instantiating the Porter Stemmer, and implementing the text preprocessing steps.
Here is the complete code:
| |
Explanation:
stop_wordsis defined using NLTK’s predefined English stop words.stemmeris instantiated using NLTK’sPorterStemmer.- In the
clean_textfunction:- The text is converted to lowercase.
- Email addresses are removed.
- URLs are removed.
- Punctuation and special characters are removed.
- Digits are removed.
- Extra spaces are removed.
- The text is processed by the
remove_stopwords_and_stemfunction.
- In the
remove_stopwords_and_stemfunction:- The text is tokenized using
word_tokenize. - Stop words are removed, and the remaining words are stemmed using a list comprehension.
- The text is tokenized using
- The first article from the 20 newsgroups dataset is processed and displayed.
output
| |
lesson4
Brace yourself for an out-of-this-world journey through text classification using n-grams! 🚀 We’re getting closer to mastering this skill, and I’m right here to navigate this adventure with you. Keep going, space explorer!
Topic Overview and Goal
Hello, and welcome to today’s lesson on n-grams! If you’ve ever wondered how language models or text classifiers can understand the context or sequence in text, it’s usually courtesy of our today’s hero — n-grams. In this lesson, we’ll delve into the magic of n-grams and how essential they prove in processing textual data. Specifically, we’ll learn how to create n-grams from text data using Python, covering unigrams and bigrams.
Topic Overview and Goal
Hello, and welcome to today’s lesson on n-grams! If you’ve ever wondered how language models or text classifiers can understand the context or sequence in text, it’s usually courtesy of our today’s hero — n-grams. In this lesson, we’ll delve into the magic of n-grams and how essential they prove in processing textual data. Specifically, we’ll learn how to create n-grams from text data using Python, covering unigrams and bigrams.
What are n-grams?
In Natural Language Processing, when we analyze text, it’s often beneficial to consider not only individual words but sequences of words. This approach helps to grasp the context better. Here is where n-grams come in handy.
An n-gram is a contiguous sequence of n items from a given sample of text or speech. The ’n’ stands for the number of words in the sequence. For instance, in “I love dogs,” a 1-gram (or unigram) is just one word, like “love.” A 2-gram (or bigram) would be a sequence of 2 words, like “I love” or “love dogs”.
N-grams help preserve the sequential information or context in text data, contributing significantly to many language models or text classifiers.
Preparing Data for n-Grams Creation
Before we can create n-grams, we need clean, structured text data. The text needs to be cleaned and preprocessed into a desirable format, after which it can be used for feature extraction or modeling.
Here’s an already familiar code where we apply cleaning on our text, removing stop words and stemming the remaining words. These steps include lower-casing words, removing punctuations, useless words (stopwords), and reducing all words to their base or stemmed form.
| |
Creating n-grams with Python: Setting up the Vectorizer
Python’s sklearn library provides an accessible way to generate n-grams. The CountVectorizer class in the sklearn.feature_extraction.text module can convert a given text into its matrix representation and allows us to specify the type of n-grams we want.
Let’s set up our vectorizer as a preliminary step towards creating n-grams:
| |
The ngram_range=(1, 2) parameter instructs our vectorizer to generate n-grams where n ranges from 1 to 2. So, the CountVectorizer will generate both unigrams and bigrams. If we wanted unigrams, bigrams, and trigrams, we could use `ngram_range=(1, 3
Creating n-grams with Python: Applying the Vectorizer
Now that we’ve set up our n-gram generating machine let’s use it on some real-world data.
| |
Applying the vectorizer to our cleaned text data will create the n-grams:
| |
The output of the above code will be:
| |
The shape of X is (100, 16246), indicating we have a high-dimensional feature space. The first number, 100, represents the number of documents or records in your dataset (here, it’s 100 as we limited our fetching to the first 100 records of the dataset), whereas 16246 represents the unique n-grams or features created from all the 100 documents.
By printing features[100:111] we get a glance into our features where each string represents an n-gram from our cleaned text data. The returned n-grams ['accid figur', 'accid worri', 'accomod', ...] include both unigrams (single words like accomod, account) and bigrams (two-word phrases like accid figur, accid worri).
As you can see, generating n-grams adds a new level of complexity to our analysis, as we now have multiple types of features or tokens - unigrams and bigrams. You can experiment with the ngram_range parameter in CountVectorizer to include trigrams or higher-level n-grams, depending on your specific context and requirements. Remember, each choice will have implications for the complexity and interpretability of your models, and it’s always a balance between the two.
Lesson Summary
Congratulations, you’ve finished today’s lesson on n-grams! We’ve explored what n-grams are and their importance in text classification. We then moved on to preparing data for creating n-grams before we dived into generating them using Python’s CountVectorizer class in the sklearn library.
Now, it’s time to get hands-on. Try generating trigrams or 4-grams from the same cleaned newsgroups data and notice the differences. Practicing these skills will not only reinforce the concepts learned in this lesson but also enable you to understand when and how much context is needed for certain tasks.
As always, happy learning!
「Practice1」
Excellent work, Space Voyager! Now, let’s deepen our understanding of n-grams in Python. Modify the ngram_range parameter in CountVectorizer in the starter code to generate only bigrams and trigrams, instead of unigrams, bigrams, and trigrams. Change ngram_range from (1, 3) to (2, 3). Display the output and observe the differences.
| |
Here is the modified code to generate only bigrams and trigrams using CountVectorizer:
| |
Output:
| |
Key changes made:
- Modified
ngram_rangefrom(1, 3)to(2, 3)in the CountVectorizer initialization to generate only bigrams and trigrams - Printed out the first 20 features from
feature_namesto show some example bigrams and trigrams
By changing the ngram_range to (2, 3), the output now contains only bigram and trigram features, without any unigrams. This allows capturing more context by considering pairs and triplets of words together[1][2].
The bigrams and trigrams can provide additional insights compared to just unigrams. For example, “new york” as a bigram has a different meaning than the individual unigrams “new” and “york”[4]. Trigrams extend this to three word phrases.
However, using higher-order n-grams also increases the dimensionality of the feature space substantially[3]. So there is a tradeoff between capturing more context with longer n-grams versus the resulting size of the vocabulary.
Citations: [1] https://stackoverflow.com/questions/24005762/understanding-the-ngram-range-argument-in-a-countvectorizer-in-sklearn [2] https://www.ibm.com/reference/python/countvectorizer [3] https://maartengr.github.io/BERTopic/getting_started/vectorizers/vectorizers.html [4] https://pages.github.rpi.edu/kuruzj/website_introml_rpi/notebooks/08-intro-nlp/03-scikit-learn-text.html [5] https://kavita-ganesan.com/how-to-use-countvectorizer/
Let’s deepen our understanding of n-grams in Python by modifying the ngram_range parameter in CountVectorizer. We’ll change it to generate only bigrams and trigrams, instead of unigrams, bigrams, and trigrams. Here’s the complete code including the necessary modifications and output display:
| |
In this script:
- Data Cleaning: We clean and preprocess the text data by removing email addresses, URLs, punctuation, special characters, and digits. We also remove stop words and apply stemming.
- CountVectorizer: We set the
ngram_rangeto(2, 3)in theCountVectorizer, which will generate only bigrams and trigrams. - Output Display: The transformed data is converted into a DataFrame to display the bigrams and trigrams.
Running this code will generate a feature matrix where the features are bigrams and trigrams extracted from the text data. You will be able to see the differences compared to when unigrams are included in the feature set.
「Practice」
For the next task, we will simplify our data input and use a simple short string. Through this, we will generate bigrams and trigrams from it. This will give us a more immediate understanding of generating n-grams. Can you fill in the blanks (____) to get the code up and running? Remember, we need to use CountVectorizer to generate bigrams and trigrams. After that, display the features created.
| |
Here’s the completed code with the necessary blanks filled in to generate bigrams and trigrams using CountVectorizer:
| |
In this code:
CountVectorizer(ngram_range=(2, 3))sets up the vectorizer to create bigrams (2-grams) and trigrams (3-grams).- The
fit_transformmethod is applied to the input text to create the n-grams. - The shape of the resulting matrix
Xis printed, as well as the feature names, which represent the bigrams and trigrams generated from the text.
[!tip]
Practice 3：Generating Bigrams and Trigrams from Two Texts
Bravo, Space Voyager! Now let’s try generating n-grams again, but with slightly different text. However, this time, the fit_transform function and its invoker have been removed. You need to fill in the blanks (____) to get the code running successfully.
The fit_transform function takes input as a list of texts. It will transform the texts into a matrix of token counts and return a sparse representation of this matrix, which we are storing as X. The fit_transform function is convenient when we want to learn a vocabulary dictionary and return document-term matrix at the same time.
For this exercise, we’ve made it a bit easier and more meaningful. We’re only generating unigrams (individual words) this time, meaning n is equal to 1.
The text we’re using this time is not a proper English sentence but a list of stemmed words to better illustrate how n-grams work on a finer level.
| |
解释
Sure, let’s complete the code snippet to use CountVectorizer to generate unigrams and transform the input texts into a matrix of token counts.
Here’s the completed code:
| |
In this snippet:
- The
CountVectorizeris set up to generate unigrams (n-gram range of 1, 1). - The
fit_transformmethod ofvectorizeris applied to the list of texts[text1, text2]to create the document-term matrixX. - The shape of
X, the feature names (unigrams), and the dense representation of the matrixXare printed out.
This will output the shape of the matrix, the features (unique unigrams), and the matrix itself, showing the counts of each unigram in the given texts.
「Practice」
You are doing an excellent job, Galactic Pioneer!
Now, let’s dive a bit deeper into creating bigrams. Set up the CountVectorizer to generate bigrams, then apply it to the cleaned data. Once you accomplish this, you’ll need to display the shape of X, the feature names, and the matrix X. You can do it!
| |
解释
To generate bigrams from the given cleaned data, we need to use CountVectorizer from the sklearn.feature_extraction.text module with the parameter ngram_range set to (2, 2). This will configure the CountVectorizer to create bigrams.
Here’s the complete code, including the setup of CountVectorizer to generate bigrams, applying it to the cleaned data, and then displaying the shape of X, the feature names, and the matrix X.
| |
Explanation
- Import Libraries: Necessary libraries are imported, including
CountVectorizer,stopwords,word_tokenize, andPorterStemmer. - Clean Text Function: A function
clean_textis defined to preprocess and clean the input text by removing email addresses, URLs, punctuation, digits, and extra spaces. It also converts text to lowercase and stems the words, removing stopwords. - Clean and Preprocess: Four short sentences are defined, cleaned, and preprocessed using the
clean_textfunction. - CountVectorizer for Bigrams: A
CountVectorizeris set up withngram_range=(2, 2)to generate bigrams only. - Apply CountVectorizer: The vectorizer is applied to the cleaned data to create a bigram frequency matrix
X. - Display Results: The shape of
X, the feature names (bigrams), and the matrixXitself are printed.
This code effectively preprocesses the text, generates bigrams, and displays the results as required. 好的，我来用三年级儿童能理解的方式解释这个过程！
清理和处理文本
我们先要清理和处理这些句子，就像整理书桌一样，把不需要的东西清理掉。我们有一些句子：
- 我爱学习数据科学
- 科学是我热爱的领域
- 算法让我着迷
- 工作在数据算法上是一种快乐
我们做这些清理工作：
- 变成小写字母：把所有大写字母变成小写。
- 去掉多余的东西：比如去掉邮件地址、网址、标点符号、数字和多余的空格。
- 去掉不需要的词：比如“的”、“是”等这些常用词。
- 词根化：把不同形式的词变成它们的基本形式，比如“学习”和“学”就变成一样的。
创建二元组（bigrams）
接下来，我们要把句子变成一对一对的词，比如“数据科学”就变成了一个二元组。就像我们把糖果配对成糖果对一样。
我们用一个工具叫CountVectorizer来做这个工作。这个工具会帮助我们找到所有句子里的词对，并数一数每个词对出现了多少次。
结果
我们把这些清理后的句子用CountVectorizer工具处理，得到这些结果：
- 形状：就像我们把所有的糖果对放在一个大盒子里，看一看这个盒子有多大。
- 特征（词对）：我们列出所有的词对，看看有哪些词对，比如“数据 科学”、“科学 热爱”等。
- 矩阵：我们得到一个表格，告诉我们每个句子里有多少次出现了这些词对。
完整代码
下面是我们完整的代码，你可以想象这是一个超级机器人，帮我们做所有这些工作：
| |
希望这个解释对你有帮助！这样我们就可以用这些工具来分析和理解文本中的词对啦！
Introduction and Overview
引言与概述
Ready for our next lesson? Today, we’re delving into quantiles and the Interquartile Range (IQR). Quantiles divide our data into equal parts, and the IQR reveals where half of our data lies. These tools aid us in understanding the distribution of our data and in identifying outliers. With Python’s pandas and NumPy libraries, we’ll explore how to calculate these measures.
准备好下一课了吗？今天，我们将深入探讨分位数和四分位距 (IQR)。分位数将我们的数据分成相等的部分，而 IQR 揭示了我们数据的一半位于何处。这些工具帮助我们理解数据的分布并识别异常值。我们将使用 Python 的 pandas 和 NumPy 库来探索如何计算这些指标。
Defining Quantiles 分位数的定义
Quantiles segment data into equal intervals. For example, when we divide a group of student grades into four equal parts, we employ quartiles (Q1 - 25th percentile, Q2 - 50th percentile or median, and Q3 - 75th percentile).
分位数将数据分割成相等的区间。例如，当我们将一组学生成绩分成四个相等的部分时，我们使用的是四分位数（Q1 - 第 25 百分位数，Q2 - 第 50 百分位数或中位数，以及 Q3 - 第 75 百分位数）。
Understanding the Interquartile Range
理解四分位距
The Interquartile Range (IQR) shows where half of our data lies. It’s resistant to outliers; for instance, when analyzing salaries, the IQR omits extreme values, thereby depicting the range where most salaries fall.
四分位距（IQR）显示了我们数据中一半数据的位置。它不受异常值的影响；例如，在分析工资时，IQR 会忽略极端值，从而描述大多数工资所在的范围。
Calculating Quantiles with Python
使用 Python 计算分位数
Python’s NumPy function, percentile(), calculates quantiles.
Python 的 NumPy 函数 percentile() 用于计算分位数。
Quantiles are essentially just cuts at specific points in your data when it’s sorted in ascending order. The first quartile (Q1) is the point below which 25% of the data falls, while the third quartile (Q3) is the point below which 75% of the data falls. The second quartile or the median is the mid-point of the data when it’s sorted in ascending order.
分位数本质上是在按升序排序的数据中特定点的切割。第一个四分位数 (Q1) 是指低于该点的数据占 25%，而第三个四分位数 (Q3) 是指低于该点的数据占 75%。第二个四分位数或中位数是数据按升序排序时的中间点。
These values are important in identifying the spread and skewness of your data. Let’s consider a dataset of student scores:
这些值对于确定数据的离散程度和偏度非常重要。让我们考虑一个学生分数数据集：
Python
CopyPlay
1import numpy as np 2 3scores = np.array([76, 85, 67, 45, 89, 70, 92, 82]) 4 5# Calculate median 6median_w1 = np.percentile(scores, 50) 7print(median_w1) # Output: 79.0 8# Check if it is the same as median 9median_w2 = np.median(scores) 10print(median_w2) # Output 79.0 11 12# Calculate Q1 and Q3 13Q1 = np.percentile(scores, 25) 14print(Q1) # Output: 69.25 15Q3 = np.percentile(scores, 75) 16print(Q3) # Output: 86.0
Here, percentile() is used to calculate the 1st, 2nd and 3rd quartiles. When we input 25, the function gives us the value below which 25% of the data lies, i.e., the first quartile Q1. Similarly, when we input 75, it gives the third quartile Q3. The 50th percentile is the median of the dataset.
这里， percentile() 被用来计算第一、第二和第三四分位数。当我们输入 25 时，函数给出的是数据中 25%低于该值的值，即第一四分位数 Q1。同样，当我们输入 75 时，它给出的是第三四分位数 Q3。第 50 个百分位数是数据集的中位数。
Calculating the Interquartile Range with Python
使用 Python 计算四分位距
The Interquartile Range (IQR) is computed as Q3 - Q1.
四分位距（ IQR ）计算公式为 Q3 - Q1 。
| |
The IQR represents the range within which the middle half of the scores fall. It exposes potential outliers, defined as values that either lie below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR. Multiplying the IQR by 1.5 roughly sets a boundary that encapsulates 99.3% of the data assuming a normal distribution. So anything outside this range could be viewed as potential outliers.
IQR 表示一半数据所在的范围。它揭示了潜在的异常值，定义为低于 Q1 - 1.5 * IQR 或高于 Q3 + 1.5 * IQR 的值。将 IQR 乘以 1.5 大致设定了一个边界，在假设数据呈正态分布的情况下，该边界包含了 99.3 %的数据。因此，超出此范围的任何数据点都可能被视为潜在的异常值。
This boundary of 1.5 times the IQR is a generally accepted rule of thumb and helps to balance between being overly sensitive to slight deviations in the data versus not being sensitive enough to detect potential anomalies or outliers. This rule is particularly useful when data is large and complex when it’s hard to discern outliers just by observation.1.5 倍 IQR 的边界是一条普遍接受的经验法则，它有助于在对数据的轻微偏差过于敏感和对检测潜在异常值或离群值不够敏感之间取得平衡。当数据量大且复杂，仅凭观察难以识别异常值时，这条规则特别有用。
Finding Outliers 查找异常值
Let’s select and print out all the outliers using the rule above. We will apply NumPy’s boolean selection, which works just fine with pandas:
让我们使用上述规则选择并打印出所有异常值。我们将应用 NumPy 的布尔选择，它与 pandas 可以很好地配合使用：
| |
Summary and Look Ahead 总结与展望
Congratulations! You’ve learned about two key statistical measures: quantiles and the Interquartile Range, as well as how to calculate them using Python.
恭喜！您已经学习了两个关键的统计指标：分位数和四分位距，以及如何使用 Python 计算它们。
In the next lesson, we’ll practice these concepts; prepare for some hands-on exercises. Practice aids in mastering these concepts. Let’s get started. Are you ready for the next lesson? Happy learning!
下一课我们将练习这些概念，准备好进行一些实践练习。练习有助于掌握这些概念。让我们开始吧。你准备好下一课了吗？祝你学习愉快！
Introduction
Welcome to our lesson on Named Entity Recognition! Today, we’ll be diving deep into the world of NLP and discovering how we can identify informative chunks of text, namely “Named Entities”. The goal of this lesson is to learn about Part of Speech (POS) tagging and Named Entity Recognition (NER). By the end, you’ll be able to gather specific types of data from text and get a few steps closer to mastering text classification.
lesson
Introduction 引言
Welcome to our lesson on Named Entity Recognition! Today, we’ll be diving deep into the world of NLP and discovering how we can identify informative chunks of text, namely “Named Entities”. The goal of this lesson is to learn about Part of Speech (POS) tagging and Named Entity Recognition (NER). By the end, you’ll be able to gather specific types of data from text and get a few steps closer to mastering text classification.
欢迎来到我们的命名实体识别课程！今天，我们将深入 NLP 的世界，探索如何识别信息丰富的文本块，即“命名实体”。本课程的目标是学习词性 (POS) 标注和命名实体识别 (NER)。在本课程结束时，您将能够从文本中收集特定类型的数据，并向掌握文本分类迈进几步。
What is Named Entity Recognition?
命名实体识别是什么？
Imagine we have a piece of text and we want to get some quick insights. What are the main subjects? Are there any specific locations or organizations being talked about? This is where Named Entity Recognition (NER) comes in handy.
假设我们有一段文本，我们想快速了解它。主要主题是什么？有没有提到具体的地点或组织？这就是命名实体识别 (NER) 的用武之地。
In natural language processing (NLP), NER is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as names of persons, organizations, locations, expressions of times, quantities, monetary values, and percentages.
在自然语言处理（NLP）中，命名实体识别（NER）是信息提取的一个子任务，旨在定位文本中出现的命名实体，并将其分类到预先定义的类别中，例如人名、组织机构名、地点、时间表达式、数量、货币值和百分比。
For instance, consider the sentence: “Apple Inc. is planning to open a new store in San Francisco.” Using NER, we could identify that “Apple Inc.” is an organization and “San Francisco” is a location. Such information can be incredibly valuable for numerous NLP tasks.
例如，考虑这句话：“苹果公司计划在旧金山开设一家新店。” 使用 NER，我们可以识别出“苹果公司”是一个组织，“旧金山”是一个地点。 这些信息对于众多 NLP 任务来说非常宝贵。
Part of Speech (POS) Tagging
词性标注 (POS)
Every word in a sentence has a particular role. Some words are objects, some are verbs, some are adjectives, and so on. Tagging these parts of speech, or POS tagging, can be a critical component to many NLP tasks. It can help answer many questions, like what are the main objects in a sentence, what actions are being taken, and what’s the context of these actions?
句子中的每个词都有特定的词性。有些词是宾语，有些词是动词，有些词是形容词，等等。对这些词性进行标记，或者说词性标注，是许多自然语言处理任务的关键组成部分。它可以帮助回答许多问题，例如句子中的主要宾语是什么，正在采取什么行动，以及这些行动的背景是什么？
Let’s start with a sentence example: “Apple Inc. is planning to open a new store in San Francisco.” We are going to use NLTK‘s pos_tag function to tag the part of speech for each word in this sentence.
让我们从一个例句开始：“苹果公司计划在旧金山开设一家新店。”我们将使用 NLTK 的 pos_tag 函数来标记这个句子中每个词的词性。
| |
The output of the above code will be:
以上代码的输出将是：
| |
Here, every word from our sentence gets tagged with a corresponding part of speech. This is the first step towards performing Named Entity Recognition.
在这里，我们句子中的每个词都被标记了相应的词性。这是进行命名实体识别（NER）的第一步。
Named Entity Recognition with NLTK
使用 NLTK 进行命名实体识别
Now, what about Named Entity Recognition? Well, Named Entity Recognition (or NER) can be considered a step beyond regular POS tagging. It groups together one or more words that signify a named entity such as “San Francisco” or “Apple Inc.” into a single category, i.e., location or organization in this case.
那么，命名实体识别呢？命名实体识别（NER）可以被视为比常规词性标注更进一步的技术。它将表示命名实体的一个或多个单词（例如“旧金山”或“苹果公司”）归类到单个类别中，在本例中分别是地点或组织。
We can use the ne_chunk function in NLTK to perform NER on our POS-tagged sentence, like so:
我们可以使用 NLTK 中的 ne_chunk 函数对我们 POS 标注的句子执行 NER，如下所示：
| |
The output of the above code will be:
以上代码的输出将是：
| |
Let’s break down this output:
让我们分析一下这个输出：
- The ‘S’ at the beginning signifies the start of a sentence.
句首的“S”表示一个句子的开始。 - Words inside paretheses, prefixed with labels such as PERSON, ORGANIZATION, or GPE are recognized named entities. For example, ‘(PERSON Apple/NNP)’ indicates that ‘Apple’ is recognized as a named entity representing a Person and ‘Apple’ has been POS tagged as ‘NNP’ (Proper Noun, Singular).
括号中的词语，如果带有诸如 PERSON、ORGANIZATION 或 GPE 等标签，则表示识别出的命名实体。例如，’(PERSON Apple/NNP)’ 表示“Apple”被识别为代表人物的命名实体，并且“Apple”已被词性标注为“NNP”（专有名词，单数）。- Words outside parentheses are not recognized as part of a named entity but are part of the sentence and each of them is associated with a POS tag. For instance, ‘is/VBZ’ means that ‘is’ is recognized as a verb in present tense, 3rd person singular form.
圆括号外的单词不被识别为命名实体的一部分，而是句子的一部分，并且每个单词都与一个词性标签相关联。例如，“is/VBZ”表示“is”被识别为现在时、第三人称单数形式的动词。 - ‘(GPE San/NNP Francisco/NNP)’ indicates that ‘San Francisco’, a two-word entity, is recognized as a geopolitical entity, such as a city, state, or country.
‘(GPE San/NNP Francisco/NNP)’ 表示“旧金山”这个由两个词组成的实体被识别为一个地缘政治实体，例如城市、州或国家。 While Named Entity Recognition offers richer insights than simple POS tagging, it might not always be perfectly accurate due to the ambiguity and context-dependent nature of language. Despite this, it’s a powerful tool in any NLP practitioner’s arsenal.
虽然命名实体识别比简单的词性标注提供了更丰富的见解，但由于语言的歧义性和语境依赖性，它可能并不总是完全准确。尽管如此，它仍然是任何自然语言处理从业者武器库中的有力工具。
- Words outside parentheses are not recognized as part of a named entity but are part of the sentence and each of them is associated with a POS tag. For instance, ‘is/VBZ’ means that ‘is’ is recognized as a verb in present tense, 3rd person singular form.
Applying PoS Tagging and NER to a Real Dataset
将词性标注和命名实体识别应用于真实数据集
Examining these NLP techniques in action on larger, more complex datasets allows us to understand the power of Natural Language Processing better. To this end, let’s use POS tagging and Named Entity Recognition on a real-world dataset - the 20 Newsgroups dataset.
在更大、更复杂的数据集上考察这些 NLP 技术的实际应用，可以让我们更好地理解自然语言处理的强大功能。为此，让我们在真实世界的数据集——20 Newsgroups 数据集——上使用词性标注和命名实体识别技术。
| |
Here’s the output you can expect:
请提供您需要翻译的文本内容。我将尽力将其准确地翻译成简体中文，并保持原文的学术语气，不添加任何解释。
| |
As you can see, even when we’re working with a slimmed-down text input, both POS tagging and NER deliver valuable insights. We’ve applied these techniques to just a portion of a complex, real-world dataset, demonstrating how NLP can uncover important information from vast amounts of textual data. This highlights the critical role NLP plays in fields ranging from data analysis to AI and machine learning.
正如您所见，即使我们处理的是精简的文本输入，词性标注和命名实体识别也能提供有价值的见解。我们已将这些技术应用于复杂、真实的数据库的一部分，展示了自然语言处理如何从海量文本数据中发现重要信息。这凸显了自然语言处理在从数据分析到人工智能和机器学习等各个领域的关键作用。
Lesson Summary and Practice
课程总结与练习
In this lesson, we have covered Part of Speech (POS) tagging, Named Entity Recognition (NER), and even applied these techniques to a real-world dataset! These concepts are fundamental to text preprocessing in Natural Language Processing (NLP). Having a grasp over these will allow you to approach more advanced topics in NLP with ease.
在本课中，我们学习了词性标注 (POS)、命名实体识别 (NER)，甚至将这些技术应用到了真实的数据集！这些概念是自然语言处理 (NLP) 中文本预处理的基础。掌握这些将使您能够轻松地处理 NLP 中更高级的主题。
You have the upcoming tasks to practice these techniques, reinforcing your understanding and improving your Natural Language Processing skills. Let’s head onto them and keep learning! Practice is key when it comes to mastering these techniques. Enjoy the hands-on session. Keep Learning!
你接下来将要完成的任务可以帮助你练习这些技巧，巩固理解并提升自然语言处理技能。让我们开始吧，持续学习！熟练掌握这些技巧的关键在于练习。享受实践的过程吧！持续学习！
「Practice」
Great job! Now, let’s shift our focus to another set of student scores, this time in Math. The given code calculates upper outlier bound, using the Interquartile Range (IQR) method, described in the lesson.
Your goal is to update code to calculate lower outlier bound instead, using the same principle.
| |
解释 To update the given code to calculate the lower outlier bound using the Interquartile Range (IQR) method, we need to follow these steps:
- Calculate the first quartile (Q1) and the third quartile (Q3) for the math scores.
- Calculate the Interquartile Range (IQR) by subtracting Q1 from Q3.
- Use the IQR to calculate the lower outlier bound by subtracting (1.5 \times \text{IQR}) from Q1.
Here is the updated code:
| |
Explanation:
- We first calculate the first quartile ((Q1)) and the third quartile ((Q3)) using
np.percentile. - Then, we find the Interquartile Range ((IQR)) by subtracting (Q1) from (Q3).
- Finally, we calculate the lower outlier bound by subtracting (1.5 \times \text{IQR}) from (Q1), which is the standard method to detect lower outliers.
This code will output the lower outlier bound for the given math scores.
「Practice」
Great work, Space Voyager!
Now, change the sentence in the provided code to “Microsoft Corporation aims to reach 95% global coverage by 2025.” and observe how the named entities identified by the Named Entity Recognition (NER) change.
Keep advancing!
| |
To change the sentence and observe how the named entities identified by the Named Entity Recognition (NER) change, we’ll modify the given sentence to “Microsoft Corporation aims to reach 95% global coverage by 2025.” and then run the NER process.
Here’s the updated code:
| |
The expected output will show the named entities identified by NER. Let’s break down the process and what changes we can expect:
- Tokenization: The sentence will be split into individual words (tokens).
- POS Tagging: Each token will be tagged with its part of speech.
- NER: The part-of-speech tagged tokens will be analyzed to identify named entities such as organizations, locations, and dates.
The initial sentence, “Apple Inc. planned to open a new store in San Francisco in 82% of time by 2020,” identifies “Apple Inc.” as an organization, “San Francisco” as a location, and “2020” as a date.
For the updated sentence, “Microsoft Corporation aims to reach 95% global coverage by 2025,” the expected named entities are:
- “Microsoft Corporation” as an organization.
- “2025” as a date.
Running the code will confirm the exact output. Here’s a mock-up of what we can expect:
| |
This output indicates that “Microsoft Corporation” is recognized as an organization, and “2025” is recognized as a date, as expected. The NER process using ne_chunk from NLTK helps in identifying and categorizing these entities.
「Practice」
Excellent work, Space Explorer!
Let’s reinforce your learning by inserting the missing code. Fill in the appropriate functions to perform tokenization and POS tagging. Use nltk’s powerful tools to achieve this. Happy learning!from nltk import ____, word_tokenize
| |
解释 Certainly, here is the completed code with the appropriate functions from the NLTK library to perform tokenization and POS tagging:
| |
Explanation:
Importing the necessary functions:
pos_tag: This function is used to perform part-of-speech (POS) tagging.word_tokenize: This function is used to split the sentence into individual words or tokens.
Tokenization:
tokens = word_tokenize(sample_sentence): This line splits the given sentence into tokens.
POS Tagging:
pos_tags = pos_tag(tokens): This line assigns POS tags to each token.
Printing the POS tags:
print(pos_tags): This prints out the tokens along with their corresponding POS tags.
When you run this code, you should see an output where each word in the sentence is paired with its POS tag, like this:
| |
Explanation of POS tags in the output:
- DT: Determiner
- JJ: Adjective
- NN: Noun, singular or mass
- VBZ: Verb, 3rd person singular present
- IN: Preposition or subordinating conjunction
This way, the code effectively tokenizes the sentence and performs POS tagging using NLTK’s powerful tools.
lesson
Introduction
Hello and welcome! Today’s lesson will introduce a crucial component of text feature engineering: tokenization. Used in text classification, tokenization is a pre-processing step that transforms raw text into units of meaning known as tokens. By breaking down text into these consumable pieces, we can provide feeding material for machine learning models to understand the text better. Our goal in this lesson is to apply tokenization on a raw text dataset (IMDB movie review dataset) and understand how it can be beneficial in the process of text classification.
Understanding the Concept and Importance of Text Tokenization
Text tokenization is a type of pre-processing step where a text string is split up into individual units (tokens). In most cases, these tokens are words, digits, or punctuation marks. For instance, consider this text: “I love Python.” After tokenization, this sentence is split into ['I', 'love', 'Python', '.'], with each word and punctuation mark becoming a separate token.
Text tokenization plays a foundational role in text classification and many Natural Language Processing (NLP) tasks. Consider the fact that most machine learning algorithms prefer numerical input. But when dealing with text data, we can’t feed raw text directly into these algorithms. This is where tokenization steps in. It breaks down the text into individual tokens, which can then be transformed into some numerical form (via techniques like Bag-of-Words, TF-IDF, etc.). This transformed form can then be processed by the machine learning algorithms.
Applying Tokenization on a Text Example Using NLTK
Before we tackle our dataset, let’s understand how tokenization works with a simple example. Python and the NLTK (Natural Language Toolkit) library, a comprehensive library built specifically for NLP tasks, make tokenization simple and efficient. For our example, suppose we have a sentence: “The cat is on the mat.” Let’s tokenize it:
| |
The output of the above code will be:
| |
Text Classification Dataset Overview
For the purpose of this lesson, we’ll use the IMDB movie reviews dataset (provided in the NLTK corpus). This dataset contains movie reviews along with their associated binary sentiment polarity labels. The core dataset has 50,000 reviews split evenly into 25k for training and 25k for testing. Each set has 12.5k positive and 12.5k negative reviews. However, for the purpose of these lessons, we will focus on using the first 100 reviews.
It’s important to note that the IMDB dataset provided in the NLTK corpus has been preprocessed. The text is already lowercased, and common punctuation is typically separated from the words. This pre-cleaning makes the dataset well-suited for the tokenization process we’ll be exploring.
Let’s get these reviews and print a few of them:
| |
Note that we’re only printing the first 260 characters of the first review to prevent lengthy output.
The output of the above code will be:
| |
Applying Tokenization on the Dataset
Now it’s time to transform our data. For this, we will apply tokenization on all our 100 movie reviews.
| |
So, what changes did tokenization bring to our data? Each review, which was initially a long string of text, is now a list of individual tokens (words, punctuation, etc), which collectively represent the review. In other words, our dataset evolved from being a list of strings to being a list of lists.
| |
The output of the above code will be:
| |
Lesson Summary and Next Steps
Well done! Today, you learned about the fundamental concept of text tokenization and its importance in text classification. You also applied tokenization to the IMDB movie reviews dataset using Python and NLTK. Your text data is now effectively transformed into a form that machine learning models can digest better.
As you advance in the course, you will refine this dataset further for your text classification objectives. We are laying the foundation one brick at a time, and tokenization was a sturdy one! Upcoming lessons will build upon this understanding. You’ll harness this tokenized data to generate Bag-of-Words representations, implement TF-IDF representations, handle sparse features, and apply dimensionality reduction.
Remember, practice consolidates learning. Make sure to reinforce your knowledge by practicing the code samples and applying these concepts contextually. Don’t forget to use your creativity to manipulate codes and see the outcomes. Happy learning!
「Practice」
Good job, Space Voyager! The existing code is intended to tokenize a movie review and filter out punctuation, leaving only word tokens. However, there’s a bug causing it to act incorrectly.
干得好，太空旅行者！现有代码旨在将电影评论标记化并过滤掉标点符号，仅保留单词标记。但是，其中存在一个错误，导致其行为不正确。
Can you locate the error and adjust the code to correctly execute the task? After your revision, the code should tokenize the first movie review from the dataset, omit any punctuation, and print the first 10 word tokens.
| |
解释 你已经很接近正确答案了！代码中只有一个小错误需要修正。让我们仔细看看这行代码：
| |
这行代码使用列表推导式来过滤标记，但条件 if not token.isalpha() 的逻辑是相反的。isalpha() 方法检查一个字符串是否只包含字母字符。通过使用 not，它实际上保留了非字母的标记（如标点符号），而丢弃了所有单词标记。
为了修复这个问题，我们需要移除 not 关键字，像这样：
| |
现在，条件 if token.isalpha() 将只保留字母字符的标记，有效地过滤掉标点符号。
修改后的完整代码如下：
| |
运行修改后的代码，你应该会看到前10个单词标记正确地打印出来，不包含任何标点符号。这样就成功地标记化了电影评论，并过滤掉了非单词标记。
| |
「Practice」
Hey Space Voyager, you’re doing great!
嘿，太空旅行者，你做得很好！
In the provided starter code, fill in the blanks (____) to convert a given sentence into individual tokens using the nltk library. Then, filter out the word tokens, leaving out the punctuation marks. 请提供需要翻译的完整句子，我才能帮您翻译成简体中文。目前您只提供了句子的开头 “In the p”，信息不足，无法进行翻译。
在提供的初始代码中，填写空格 ( ____ )，使用 nltk 库将给定的句子转换为单独的标记。然后，过滤掉单词标记，去掉标点符号。
| |
「Practice」
Hey Space Voyager, you’re doing great!
In the provided starter code, fill in the blanks (____) to convert a given sentence into individual tokens using the nltk library. Then, filter out the word tokens, leaving out the punctuation marks.
| |
解释
| |
Explanation:
word_tokenize: This function from NLTK breaks the text into individual words and punctuation.token.isalpha(): This method checks if the token consists only of alphabetic characters, thus effectively filtering out punctuation and numbers.
When you run this script, it will print the list of word tokens from the given text, excluding any punctuation marks.
| |
| |
根据你的要求，我已经修改了代码，将正态分布的参数从 loc=30, scale=5 调整到 loc=35, scale=10。下面是修正后的完整代码段：
| |
这段代码将生成新的天气数据，并计算其偏度和峰度，以分析数据的分布特性。如需进一步分析或调整，请告知！
「Pratice」
Great job so far, Space Voyager!
Your task now is to fill in the blanks with the correct code to load the IMDB movie reviews dataset. After loading the data, tokenize the first movie review, and print the first 10 tokens of the tokenized review.
| |
解释
| |
This script downloads the movie_reviews dataset and the necessary tokenizer (punkt). It then loads the text of the first movie review, tokenizes it, and prints the first 10 tokens. If you have any further questions or need more assistance, feel free to ask!
lesson
Introducing Bag-of-Words Representation
In the world of text analysis, transforming raw data into a format that is both computer-friendly and preserves the essential information for further processing is crucial. One of the simplest yet versatile methods to do this is the Bag-of-Words Representation, or BoW for short.
BoW is essentially a method to extract features from text. Imagine you have a big bag filled with words. These words can come from anywhere: a book, a website, or, in our case, movie reviews from the IMDB dataset. For each document or sentence, the BoW representation will contain the count of how many times each word appears. Most importantly, in this “bag,” we don’t care about the order of words, only their occurrence.
Consider this simple example with three sentences:
The cat sat on the mat.The cat sat near the mat.The cat played with a ball.
Using a BoW representation, our table would look like this:
| the | cat | sat | on | mat | near | played | with | a | ball | |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| 2 | 2 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
| 3 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 |
Each sentence (document) corresponds to a row, and each unique word is a column. The values in the cells represent the word count in the given sentence.
Illustrating Bag-of-Words with a Simple Example
We can start practising the Bag-of-Words model by using Scikit-learn CountVectorizer on the exact same three sentences:
| |
The output of the above code will be:
| |
From the output, you’ll notice that Scikit-learn CountVectorizer has done the exact thing as our previous manual process. It’s created a Bag-of-Words representation for our sentences where each row corresponds to a sentence and each column to a unique word.
Applying Bag-of-Words to Our Dataset
Now that we know what Bag-of-Words is and what it does, let’s apply it to our dataset:
| |
In the code snippet above, we utilize Python’s NLTK module to download and import the IMDB movie reviews dataset.
Next, we’ll again use Scikit-learn’s CountVectorizer to apply the BoW method to our reviews:
| |
The output of the above code will be:
| |
The output indicates that the result is a matrix where each row corresponds to a movie review and each column to a unique word. The entries in this matrix are word counts.
Understanding the Bag-of-Words Matrix and Most Used Word
Let’s decode what’s inside the bag_of_words matrix:
| |
Here, we retrieve the feature names (which are unique words in the reviews) from our CountVectorizer model. Then we get the word counts for a specific review - in our case, we chose the first one.
Subsequently, let’s find out which word in the first review occurs the most:
| |
Running the above code would output something like:
| |
The output gives away the most used word in the first review and its count. The script finds the index of the word with the highest count in the first review. Then, it uses this index to identify the corresponding word in the feature_names. This demonstrates how we can identify the most used word in a specific review using the Bag-of-Words model.
Lesson Summary
Congratulations! You’ve successfully made it through this lesson. Today, you’ve learned how to implement a significant concept in the world of text classification, the Bag-of-Words method. You’ve not only understood the theoretical aspect of it, but you also applied it on a real-world dataset using Python. You even used it to extract insights about word frequency, a crucial aspect of many text classification problems.
As we move forward in the upcoming lessons, we’ll take what you’ve learned today, build on top of it, and continue our journey to understand and apply more advanced text classification techniques. Remember, practice makes perfect, so try to apply what you’ve learned today on different text data on your own. Happy coding, and see you in the next lesson!
Start practice
「Practice」
Great job! Now, let’s play around with the text. Change the provided sentences to any three sentences of your choice. Then, observe how this affects the Bag-of-Words representation. Remember, the output should be a matrix, with each row representing one of your sentences and each column representing a unique word.
| |
解释 Certainly! I’ll modify the sentences and show you how it affects the Bag-of-Words representation. Here’s the updated code with new sentences:
| |
Let’s break down the filled-in parts:
We import
CountVectorizerfromsklearn.feature_extraction.text.We create an instance of
CountVectorizerwithvectorizer = CountVectorizer().We use
fit_transform(sentences)to both fit the vectorizer to our sentences (learning the vocabulary) and transform the sentences into a Bag-of-Words representation in one step.
When you run this code, you’ll see output similar to this:
| |
This output shows:
- The unique words (features) extracted from all sentences.
- A matrix where each row represents a sentence, and each column represents a word. The numbers in the matrix indicate how many times each word appears in each sentence.
For example, in the second sentence, ’learning’ appears twice, which is reflected in the matrix.
This Bag-of-Words representation is a fundamental technique in natural language processing and is often used as a starting point for more complex text analysis tasks. Great job on exploring this concept!
Introduction to Sparse Matrices
Hello and welcome to this lesson on “Efficient Text Data Representation with Sparse Matrices”. As you recall, in our previous lessons, we transformed raw text data into numerical features, for example, using the Bag-of-Words (BoW) or Term Frequency-Inverse Document Frequency (TF-IDF) techniques. These transformation methods often create what we call “Sparse Matrices,” an incredibly memory-efficient way of storing high-dimensional data.
Let’s break this down a bit. In the context of text data, each unique word across all documents could be treated as a distinct feature. However, each document will only include a small subset of these available features or unique words. Meaning, most entries in our feature matrix end up being 0s, hence resulting in a sparse matrix.
We’ll begin with a simple non-text matrix to illustrate sparse matrices and later connect this knowledge to our journey on text data transformation.
| |
Sparse Matrix Formats: CSR
In this section, we’ll investigate how we can handle sparse matrices in different formats including: Compressed Sparse Row (CSR), Compressed Sparse Column (CSC), and the Coordinate (COO) formats.
We’ll start with the CSR format, a common format for sparse matrices that is excellent for quick arithmetic operations and matrix vector calculations.
| |
The output of the above code will be:
| |
##Sparse Matrix Formats: CSC
Next, let’s convert our vectors matrix to the CSC format. This format, like the CSR format, also forms the backbone of many operations we perform on sparse matrices. But it stores the non-zero entries column-wise, and is especially efficient for column slicing operations.
| |
The output of the above code will be:
| |
In this Compressed Sparse Column output, the non-zero entries are stored column-wise. Essentially, CSC format is a transpose of the CSR format.
Sparse Matrix Formats: COO
Lastly, let’s convert our example to the COO format or Coordinate List format. The COO format is another useful way to represent a sparse matrix and is simpler compared to CSR or CSC formats.
| |
The output of the above code will be:
| |
In the COO format, or Coordinate format, the non-zero entries are represented by their own coordinates (row, column). Unlike CSC or CSR, the COO format can contain duplicate entries. This can be particularly useful when data is being accumulated in several passes and there might be instances where duplicate entries are generated. These duplicates are not immediately merged in the COO format, providing you with flexibility for subsequent processing like duplicate resolution.
Vectorized Operations: CSR and CSC
Sparse matrices are not just memory-efficient storage mechanisms, but they also allow us to conduct operations directly on them. Specifically, the CSR and CSC formats support these operations directly, whereas the COO format requires converting back to CSR or CSC first.
Let’s see this in practice when performing a multiplication operation.
| |
The output of the code block above will be:
| |
We can see the impressive CSR format efficiency in vectorized operations, which becomes crucial when performing calculations with large text data.
Vectorized Operations: COO
And now let’s demonstrate the process of performing the same multiplication operation on the COO format, but this time requiring conversion to CSR or CSC first.
| |
The output of the above code will be:
| |
The Connection Between Sparse Matrices and NLP
After going through the concepts and code, you might ask - what does all this have to do with NLP? Well, remember when we transformed raw text data into either a Bag-of-Words or a TF-IDF representation in the previous lessons? Each unique word across all documents was treated as a distinct feature. Given the high dimensionality and inherent sparsity of the resulting feature representation, we used sparse matrices for efficient storage.
Handling of sparse matrices becomes crucial in large NLP tasks, as they allow us to operate on large datasets while maintaining computational efficiency and optimal memory usage. Therefore, understanding these different formats of sparse matrices is an essential part of your feature engineering skills for text classification.
Lesson Summary
Congratulations! Today, you gained an insight into sparse matrices and their different formats, how they help efficiently storing and operating on high dimensional data like that of text records in NLP. You also explored the implications of implementing vectorized operations on different sparse matrix formats. Structuring your learning and understanding these formats is paramount to efficiently handle large datasets in NLP and other machine learning tasks. In the upcoming exercises, you’ll get hands-on experience with these concepts, reinforcing your understanding further. Keep up the momentum and dive into practice!
Topic Overview and Actualization
主题概述与现实化
Today, we target duplicates and outliers to clean our data for more accurate analysis.
今天，我们将针对重复数据和异常值进行清理，以便进行更准确的分析。
Understanding Duplicates in Data
理解数据中的重复项
Let’s consider a dataset from a school containing students’ details. If a student’s information appears more than once, that is regarded as a duplicate. Duplicates distort data, leading to inaccurate statistics.
考虑一个包含学生详细信息的学校数据集。如果一个学生的资讯出现多次，则被视为重复数据。重复数据会扭曲数据，导致统计数据不准确。
「Practice」
Greetings, Stellar Navigator! For this assignment, we’re focusing on model initialization and training. You will find a TODO comment in the provided starter code. Fill it in to define the Naive Bayes model and train it! You’ll be able to see the difference between your model’s prediction and the actual results visually on a scatter plot. Let’s dive in! 你好，星际领航员！本次作业的重点是模型初始化和训练。你会在提供的初始代码中找到一个 TODO 注释。请完成注释内容，定义朴素贝叶斯模型并对其进行训练！你将能够在散点图上直观地看到你的模型预测结果与实际结果之间的差异。让我们开始吧！
| |
「Practice」
亲爱的太空旅行者，你的技能再次被需要！利用你所学到的关于朴素贝叶斯模型的知识，你的任务是使用混淆矩阵评估你的模型。实现朴素贝叶斯模型，进行预测，然后使用测试数据为模型生成混淆矩阵。绘制混淆矩阵的结果以进行视觉评估。Dear Space Voyager, your skills are needed once again! Using what you’ve learned about the Naive Bayes Model, your mission is to evaluate your model using a confusion matrix. Implement the Naive Bayes model, make predictions, and then generate a confusion matrix for the model using the test data. Plot the results of the confusion matrix for visual assessment.
| |
解释
| |
好的,让我们用更简单的方式,给5岁的小朋友解释这个有趣的游戏: 1. 想象你有一大堆信件。有些是好朋友写的,有些是坏人寄来的骗人的信。 2. 我们要教一个机器人朋友分辨哪些是好信,哪些是坏信。 3. 首先,我们给机器人一些信看,告诉它哪些是好的,哪些是坏的。这就像教你分辨苹果和橘子。 4. 然后,我们给机器人一些新的信,看它能不能猜对是好信还是坏信。 5. 最后,我们画了一个有颜色的图画: - 蓝色的格子越深,说明机器人猜对的次数越多。 - 浅色的格子说明机器人有时候会猜错。 6. 如果图画上深蓝色的格子多,浅色的格子少,就说明我们的机器人朋友学得很好! 这就像教你的玩具机器人分辨好东西和坏东西。如果它学得好,就能帮我们找出那些骗人的坏信,保护我们不受骗。是不是很神奇呀?
| |
| |
Implementing Support Vector Machines for Text Classification
Let’s delve into the practical implementation of SVM for text classification using the Scikit-learn library. We are going to introduce a new Scikit-learn function, SVC(). This function is used to fit the SVM model according to the given training data.
In the following Python code, we initialize the SVC model, fit it with our training data, and then make predictions on the test dataset.
| |
The SVC function takes several parameters, with the key ones being:
C: This is the penalty parameter of the error term. It controls the trade off between smooth decision boundary and classifying training points correctly.kernel: Specifies the kernel type to be used in the algorithm. It can be ’linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’ or a callable.degree: Degree of the polynomial kernel function (‘poly’). Ignored by all other kernels.Making Predictions and Evaluating the SVM Model
进行预测和评估支持向量机模型
After building the model, the next step is to use it on unseen data and evaluate its performance. The python code for this step is shown below:
模型构建完成后，下一步是在未见过的数据上使用该模型并评估其性能。下面展示了此步骤的 Python 代码：
| |
The output of the above code will be:
以上代码的输出结果为：
| |
This output signifies that our SVM model has achieved a high accuracy, specifically 98%, in classifying messages as spam or ham, highlighting its effectiveness in text classification tasks.
此结果表明，我们的支持向量机 (SVM) 模型在将邮件分类为垃圾邮件或非垃圾邮件方面达到了 98% 的高准确率，凸显了其在文本分类任务中的有效性。
Lesson Summary and Upcoming Practice
课程总结和即将进行的练习
Congratulations on making it to the end of this lesson! You have now learned the theory behind Support Vector Machines (SVMs) and how to use them to perform text classification in Python. You’ve also learned to load and preprocess the data, build the SVM model, and evaluate its accuracy.
恭喜你完成了本课的学习！你现在已经学习了支持向量机 (SVM) 背后的理论，以及如何使用它们在 Python 中执行文本分类。你还学习了如何加载和预处理数据、构建 SVM 模型以及评估其准确性。
Remember, like any other skill, programming requires practice. The upcoming practice exercises will allow you to reinforce the knowledge you’ve acquired in this lesson. They have been carefully designed to give you further expertise in SVM and text classification. Good luck! You’re doing a great job, and I’m excited to see you in the next lesson on Decision Trees for text classification.
记住，编程和其他技能一样，都需要练习。接下来的练习将帮助你巩固在本节课中学到的知识。这些练习经过精心设计，旨在让你进一步掌握支持向量机和文本分类方面的知识。祝你好运！你做得很好，我期待在下一节关于决策树文本分类的课程中见到你。
Certainly! I’ll modify the code to use the polynomial kernel (‘poly’) instead of the linear kernel for the SVM model. Here’s the adjusted code:
| |
好的,我会尝试用简单的语言来解释这段代码,就像是在跟6岁的小朋友讲故事一样。 想象一下,我们有一个神奇的机器人朋友,它可以帮我们分辨短信是不是垃圾短信。我们要教这个机器人怎么做到这一点。这就是这段代码要做的事情。 1. 首先,我们需要给机器人一些工具。这些工具就像是机器人的眼睛和大脑,帮它看东西和思考。 2. 然后,我们给机器人一大堆短信。有些是好短信,有些是坏短信(垃圾短信)。 3. 我们把这些短信分成两部分:一部分用来教机器人,另一部分用来测试机器人学得怎么样。 4. 接下来,我们教机器人看短信。我们告诉它要注意短信里的每个字,就像你在学习认字一样。 5. 现在,我们要教机器人一个特殊的技能,叫做"多项式核"。这个技能可以帮助机器人理解更复杂的短信内容。 6. 机器人开始学习了!它仔细看每一条教它的短信,努力记住哪些是好短信,哪些是坏短信。 7. 学习完后,我们给机器人一些新的短信,看它能不能正确分辨出哪些是垃圾短信。 8. 最后,我们要给机器人打分。如果它猜对了很多,就说明它学得很好! 9. 我们把机器人的分数打印出来,看看它学得怎么样。 这就是整个过程啦!我们教会了机器人朋友一个新技能,让它可以帮我们分辨垃圾短信。是不是很神奇呀?
Topic Overview and Actualization
Hello and welcome! In today’s lesson, we dive into the world of Decision Trees in text classification. Decision Trees are simple yet powerful supervised learning algorithms used for classification and regression problems. In this lesson, our focus will be on understanding the Decision Tree algorithm and implementing it for a text classification problem. Let’s get started!
Understanding Decision Trees for Classification
Decision Trees are a type of flowchart-like structure in which each internal node represents a feature, each branch represents a decision rule, and each leaf node represents an outcome or a class label. The topmost node in a Decision Tree is known as the root node, which best splits the dataset.
Splitting is a process of dividing a node into two or more sub-nodes, and a Decision Tree uses certain metrics during this training phase to find the best split. These include Entropy, Gini Index, and Information Gain.
The advantage of Decision Trees is that they require relatively little effort for data preparation yet can handle both categorical and numeric data. They are visually intuitive and easy to interpret.
Let’s see how this interprets to our spam detection problem.
Loading and Preprocessing the Data
Before we dive into implementing Decision Trees, let’s quickly load and preprocess our text dataset. This step will transform our dataset into a format that can be input into our machine learning models. This code block is being included for completeness:
| |
With our data now prepared, let’s move on to implementing Decision Trees using the Scikit-learn library.
Implementing Decision Trees for Text Classification
In this section, we create our Decision Trees model using the Scikit-learn library:
| |
Here, we initialize the model using the DecisionTreeClassifier() class and then fit it to our training data with the fit() method.
Prediction and Model Evaluation
After our model has been trained, it’s time to make predictions on the test data and evaluate the model’s performance:
| |
Lastly, we calculate the accuracy score, which is the ratio of the number of correct predictions to the total number of predictions. The closer this number is to 1, the better our model:
| |
The output of the above code will be:
| |
This high accuracy score indicates that our Decision Tree model is performing exceptionally well in classifying messages as spam or not spam.
Lesson Summary and Practice
Great job! You’ve learned the theory of Decision Trees, successfully applied it to a text classification problem, and evaluated the performance of your model. Understanding and mastering Decision Trees is an essential step in your journey to becoming skilled in Natural Language Processing and Machine Learning. To reinforce what we’ve learned, the next step is to tackle some exercises that will give you hands-on experience with Decision Trees. This practical experience will reinforce your learning and deepen your understanding.
Looking forward to delving even deeper into natural language processing? Let’s proceed to our next lesson: Random Forest for Text Classification. Happy Learning!
「Practice1」
Great work! Now, let’s adjust the max_depth of our Decision Tree model and observe any potential changes in accuracy. Change the value of the max_depth parameter in DecisionTreeClassifier from 5 to 10. After adjusting your code, run it and compare the accuracy score to that of the original model. Good luck, Stellar Navigator!
| |
Certainly! I’ll adjust the max_depth parameter of the DecisionTreeClassifier from 5 to 10 and run the code to compare the accuracy scores. Here’s the modified code:
| |
Now, let’s run this code and compare the accuracy to the original model with max_depth=5. After running the code, we should see the accuracy score for the Decision Tree Classifier with max_depth=10. Let’s compare this to the previous accuracy score: 1. Original model (max_depth=5): 0.96 (from the previous output) 2. New model (max_depth=10): [The new accuracy score will be displayed here] By increasing the max_depth from 5 to 10, we’ve allowed the decision tree to create more complex decision boundaries. This can potentially lead to better accuracy on the training data, but it may also increase the risk of overfitting. If the new accuracy is higher, it suggests that the model benefited from the increased complexity and was able to capture more nuanced patterns in the data. However, if the accuracy is similar or lower, it might indicate that the additional depth didn’t provide significant improvements or possibly led to overfitting. To further analyze the impact of changing the max_depth, you could consider: 1. Comparing the training accuracy to the test accuracy to check for overfitting. 2. Trying different max_depth values to find the optimal depth for this dataset. 3. Implementing cross-validation to get a more robust estimate of the model’s performance. Remember, the goal is to find the right balance between model complexity and generalization ability. The optimal max_depth will depend on the specific characteristics of your dataset. Execution Error: ModuleNotFoundError Traceback (most recent call last) Cell In[3], line 6 4 from sklearn.feature_extraction.text import CountVectorizer 5 from sklearn.model_selection import train_test_split —-> 6 import datasets 7 import pandas as pd 9 # Load the dataset ModuleNotFoundError: No module named ‘datasets’
哦,我明白了。让我们试着用更简单的方式来解释这个情况: 想象你有一个很酷的玩具机器人。这个机器人可以学习做不同的事情,比如分辨苹果和橘子。 有时候,我们想让机器人变得更聪明。所以我们给它一个新的"大脑"。这个新"大脑"可以让机器人思考得更多。 但是,有趣的是,让机器人思考更多并不总是让它变得更聪明。有时候,它可能会变得有点糊涂,因为它想得太多了! 就像你玩游戏一样,有时候简单的规则反而更容易赢。机器人也是这样,有时候简单一点反而能做得更好。 所以,我们总是要试试看,到底是让机器人思考多一点好,还是思考少一点好。这就是大人们在做的事情 - 他们在帮助机器人找到最好的"思考方式",让它变得既聪明又不会糊涂。 你觉得怎么样?这样解释是不是更容易理解了呢?
lesson
Introduction to the Random Forest for Text Classification Lesson
Welcome to the lesson on Random Forest for Text Classification. As we continue our journey into the world of text classification techniques in Natural Language Processing (NLP), this lesson brings us to the powerful ensemble learning method - the Random Forest algorithm.
In this lesson, we will:
- Broaden our understanding of the Random Forest algorithm.
- Apply it using Python’s scikit-learn package, on the SMS Spam Collection dataset.
- Evaluate our model’s accuracy in classifying whether a text message is spam or not.
By the end of this lesson, you will have gained hands-on experience in implementing a Random Forest classifier, equipping you with another versatile tool in your NLP modeling toolkit.
Let the learning begin!
Introduction to the Random Forest for Text Classification Lesson
Welcome to the lesson on Random Forest for Text Classification. As we continue our journey into the world of text classification techniques in Natural Language Processing (NLP), this lesson brings us to the powerful ensemble learning method - the Random Forest algorithm.
In this lesson, we will:
- Broaden our understanding of the Random Forest algorithm.
- Apply it using Python’s scikit-learn package, on the SMS Spam Collection dataset.
- Evaluate our model’s accuracy in classifying whether a text message is spam or not.
By the end of this lesson, you will have gained hands-on experience in implementing a Random Forest classifier, equipping you with another versatile tool in your NLP modeling toolkit.
Let the learning begin!
| |
Remember, the CountVectorizer transforms the text data into vectors of token occurrence counts (also known as bag of words), which is required for processing by machine learning models. We also use a stratified train-test split to ensure a balanced representation of different classes within both our training and test data.
Random Forest Classification: Overview
Random Forest is a type of ensemble learning method, where a group of weak models work together to form a stronger predictive model. A Random Forest operates by constructing numerous decision trees during training time and outputting the class that is the mode of the classes (classification) of the individual trees.
Random Forest has several advantages over a single decision tree. Most significant among these is that by building and averaging multiple deep decision trees trained on different parts of the same training data, the Random Forest algorithm reduces the problem of overfitting.
Random Forests also handle imbalanced data well, making them a good option for our text classification task.
Implementing Random Forest Classifier with Scikit-learn
Now that we have a basic understanding of the Random Forest algorithm, let’s train our model.
| |
Here, the parameter n_estimators defines the number of trees in the forest of the model while random_state sets a seed to the random generator, ensuring that the split you generate is replicable. The random forest model inherently handles multi-class tasks, hence we don’t have to use the ‘one-vs-all’ method to extend it to multi-class.
这里，参数 n_estimators 定义了模型森林中树的数量，而 random_state 为随机生成器设置了一个种子，确保生成的划分是可复制的。随机森林模型本身就能处理多分类任务，因此我们不必使用“一对多”方法将其扩展到多分类。
Evaluating the Model 模型评估
Once our model is trained, we can use it to make predictions on our test data. By comparing these predictions against the actual labels in the test set, we can evaluate how well our model is performing. One of the most straightforward metrics we can use to achieve this is accuracy, calculated as the proportion of true results among the total number of cases examined.
模型训练完成后，我们可以用它对测试数据进行预测。通过将这些预测结果与测试集中实际标签进行比较，我们可以评估模型的性能。准确率是最直观的评估指标之一，它指的是在所有样本中预测正确的比例。
| |
The output of the above code will be:
以上代码的输出结果为：
| |
This indicates that our Random Forest model was able to accurately classify 97% of the messages in the test set as spam or ham, showcasing a high level of performance.
这表明我们的随机森林模型能够准确地将测试集中 97% 的消息分类为垃圾邮件或非垃圾邮件，展现出很高的性能水平。
Lesson Summary and Next Steps
课程总结和后续步骤
We successfully explored the Random Forest algorithm, learned how it works, and implemented it in Python to classify messages as spam or ham. Remember, choosing and training a model is just part of the machine learning pipeline. Evaluating your model’s performance, and selecting the best one, is also integral to any successful Machine Learning project.
我们成功探索了随机森林算法，学习了它的工作原理，并在 Python 中实现了它，以将消息分类为垃圾邮件或非垃圾邮件。请记住，选择和训练模型只是机器学习流程的一部分。评估模型的性能并选择最佳模型也是任何成功的机器学习项目的组成部分。
In our upcoming exercises, you will get the opportunity to apply the concepts you’ve learned and further familiarize yourself with the Random Forest algorithm. These tasks will help you solidify your understanding and ensure you are able to apply these techniques to your future data science projects. Happy learning!
在接下来的练习中，您将有机会运用所学概念，并进一步熟悉随机森林算法。这些任务将帮助您巩固理解，确保您能够将这些技术应用到未来的数据科学项目中。祝学习愉快！