IT 위키 - 사용자 기여 [ko]

RAMCloud

2025-02-01T03:35:22Z

Matei: 새 문서: '''RAMCloud''' is a distributed in-memory storage system designed for low-latency and high-throughput applications. It provides persistent storage with sub-microsecond access times by keeping all data in DRAM while ensuring durability through fast logging to disk or flash. ==Overview== RAMCloud aims to combine: *'''Low-Latency Storage:''' Data is stored entirely in DRAM for rapid access. *'''High Availability:''' Data is replicated across servers for fault tolerance. *'''Durabil...

'''RAMCloud''' is a distributed in-memory storage system designed for low-latency and high-throughput applications. It provides persistent storage with sub-microsecond access times by keeping all data in DRAM while ensuring durability through fast logging to disk or flash.
==Overview==
RAMCloud aims to combine:
*'''Low-Latency Storage:''' Data is stored entirely in DRAM for rapid access.
*'''High Availability:''' Data is replicated across servers for fault tolerance.
*'''Durability:''' Uses fast disk/flash logging to prevent data loss.
*'''Scalability:''' Can scale to thousands of nodes while maintaining low-latency access.
RAMCloud is particularly useful in environments requiring real-time data access, such as financial systems, search engines, and large-scale web applications.
==Key Features==
*'''Sub-Microsecond Latency:''' Provides faster access than traditional disk-based storage.
*'''Distributed Key-Value Store:''' Supports efficient data retrieval across a cluster.
*'''Crash Recovery in Seconds:''' Recovers lost data quickly by reloading from logs.
*'''High Scalability:''' Designed to handle petabyte-scale datasets with thousands of servers.
==How RAMCloud Works==
#'''Data Storage in DRAM:''' All active data is stored in memory for fast retrieval.
#'''Log-Structured Storage:''' Updates are written sequentially to persistent logs.
#'''Crash Recovery Mechanism:''' Lost data is restored by replaying logs across servers.
#'''Distributed Coordination:''' A master node manages metadata, while worker nodes handle data storage.
==Example Usage==
RAMCloud supports a key-value API that allows fast reads and writes:<syntaxhighlight lang="c++">
// Connect to a RAMCloud cluster
RAMCloud::Client client("tcp:host=ramcloud-cluster");

// Store a key-value pair
client.write("myTable", "key1", "Hello RAMCloud!");

// Retrieve a value
string value;
client.read("myTable", "key1", &value);
cout << "Retrieved: " << value << endl;
</syntaxhighlight>
==Comparison with Other Storage Systems==
{| class="wikitable"
!Feature!!RAMCloud!!Redis!!Apache Cassandra
|-
|'''Storage Medium'''||DRAM (with disk backup)||DRAM||Disk
|-
|'''Primary Use Case'''||Low-latency storage||Caching||Distributed database
|-
|'''Replication'''||Log-based persistence||In-memory replication||Multi-node replication
|-
|'''Fault Tolerance'''||Fast recovery via logs||Data loss risk without persistence||High availability with replication
|}
==Advantages==
*Provides ultra-low-latency storage.
*Recovers from crashes within seconds.
*Scales efficiently across large distributed clusters.
==Limitations==
*Requires large amounts of DRAM, making it expensive.
*Not suitable for workloads requiring deep historical storage.
*Limited adoption compared to more established distributed databases.
==Applications==
*'''Real-Time Analytics:''' Used in financial trading and fraud detection.
*'''Search Engine Indexing:''' Supports rapid access to large indexes.
*'''Web Applications:''' Reduces response times for latency-sensitive services.
*'''Machine Learning Serving:''' Stores feature embeddings for fast model inference.
==See Also==
*[[In-Memory Computing]]
*[[Distributed Storage]]
*[[Redis]]
*[[Apache Cassandra]]
*[[Key-Value Store]]
*[[High-Performance Computing]]
[[분류:Distributed Computing]]

Resilient Distributed Datasets

2025-02-01T03:34:54Z

Matei: 새 문서: '''Resilient Distributed Datasets (RDDs)''' are the fundamental data structure in Apache Spark that provide fault-tolerant, parallel computation on large datasets. RDDs enable efficient distributed data processing while ensuring resilience to failures. ==Overview== RDDs are immutable, distributed collections of objects that can be processed in parallel. They are designed to optimize large-scale data processing by: *'''Fault Tolerance:''' Automatically recovering lost data us...

'''Resilient Distributed Datasets (RDDs)''' are the fundamental data structure in [[Apache Spark]] that provide fault-tolerant, parallel computation on large datasets. RDDs enable efficient distributed data processing while ensuring resilience to failures.
==Overview==
RDDs are immutable, distributed collections of objects that can be processed in parallel. They are designed to optimize large-scale data processing by:
*'''Fault Tolerance:''' Automatically recovering lost data using lineage (recomputing from original data).
*'''In-Memory Processing:''' Storing intermediate results in memory to improve performance.
*'''Lazy Evaluation:''' Transformations are not executed immediately but only when an action is triggered.
*'''Partitioning:''' Data is split across nodes to allow parallel execution.
==Key Features==
*'''Immutability:''' Once created, RDDs cannot be modified; transformations create new RDDs.
*'''Lineage Tracking:''' Maintains a history of transformations to recompute lost partitions.
*'''Lazy Evaluation:''' Delays execution until an action (e.g., count, collect) is called.
*'''Fault Tolerance:''' Automatically recomputes lost partitions without replicating data.
*'''Parallel Computation:''' Distributes tasks across nodes in a Spark cluster.
==Creating RDDs==
RDDs can be created in two main ways:
#'''From an existing collection:'''
<syntaxhighlight lang="python">
data = [1, 2, 3, 4, 5]
rdd = sparkContext.parallelize(data)
</syntaxhighlight>
#'''From an external data source:'''
<syntaxhighlight lang="python">
rdd = sparkContext.textFile("hdfs://path/to/file.txt")
</syntaxhighlight>
==Transformations and Actions==
RDDs support two types of operations:
===Transformations (Lazy Evaluation)===
Transformations produce new RDDs from existing ones but do not execute immediately:
*'''map(func)''' – Applies a function to each element.
*'''filter(func)''' – Keeps elements that satisfy a condition.
*'''flatMap(func)''' – Similar to map but allows returning multiple values per input.
*'''union(rdd)''' – Merges two RDDs.
===Actions (Trigger Execution)===
Actions compute and return results or store data:
*'''collect()''' – Returns all elements to the driver.
*'''count()''' – Returns the number of elements in the RDD.
*'''reduce(func)''' – Aggregates elements using a function.
*'''saveAsTextFile(path)''' – Saves the RDD to a storage location.
==RDD Lineage and Fault Tolerance==
RDDs achieve fault tolerance through lineage tracking:
*Instead of replicating data, Spark logs the sequence of transformations.
*If a node fails, Spark recomputes lost partitions from the original dataset.
*This approach minimizes storage overhead while ensuring reliability.
==Comparison with Other Distributed Data Models==
{| class="wikitable"
!Feature!!RDDs (Spark)!!MapReduce (Hadoop)!!DataFrames (Spark)
|-
|'''Data Processing'''||In-memory||Disk-based||Optimized execution plans
|-
|'''Fault Tolerance'''||Lineage (recomputes lost data)||Replication||Lineage (like RDDs)
|-
|'''Performance'''||Fast (RAM-based)||Slow (disk I/O)||Faster (columnar storage)
|-
|'''Ease of Use'''||Low (requires functional programming)||Low (requires custom Java/Python)||High (SQL-like API)
|}
==Advantages==
*'''High Performance:''' In-memory computation reduces I/O overhead.
*'''Scalability:''' Designed to handle petabyte-scale data.
*'''Fault Tolerance:''' Efficient recovery via lineage tracking.
*'''Flexible API:''' Supports functional programming in Scala, Python, Java.
==Limitations==
*'''Complex API:''' Requires functional programming knowledge.
*'''High Memory Usage:''' Inefficient for certain workloads compared to optimized data structures like DataFrames.
*'''No Schema Optimization:''' Unlike DataFrames, RDDs do not optimize queries automatically.
==Applications==
*'''Big Data Processing:''' Used in large-scale ETL and analytics pipelines.
*'''Machine Learning:''' Supports distributed ML algorithms via [[MLlib]].
*'''Graph Processing:''' Backbone of [[GraphX]] for scalable graph analytics.
==See Also==
*[[Apache Spark]]
*[[DataFrames (Spark)]]
*[[Hadoop MapReduce]]
*[[Distributed Computing]]
*[[Big Data Processing]]
[[분류:Distributed Computing]]

Distributed Shared Memory

2025-02-01T01:53:10Z

Matei: 새 문서: '''Distributed Shared Memory (DSM)''' is a memory management architecture that enables multiple distributed systems to share a logical memory space, allowing processes on different machines to access shared data as if they were on a single system. ==Overview== Distributed shared memory provides an abstraction that simplifies parallel and distributed computing by: *'''Providing a unified memory model:''' Applications access memory as if it were shared, even though it is physicall...

'''Distributed Shared Memory (DSM)''' is a memory management architecture that enables multiple distributed systems to share a logical memory space, allowing processes on different machines to access shared data as if they were on a single system.
==Overview==
Distributed shared memory provides an abstraction that simplifies parallel and distributed computing by:
*'''Providing a unified memory model:''' Applications access memory as if it were shared, even though it is physically distributed.
*'''Reducing explicit message passing:''' Eliminates the need for developers to manually implement interprocess communication (IPC).
*'''Ensuring consistency:''' Various memory consistency models define how updates to shared data are propagated.
==Key Features==
*'''Transparency:''' Abstracts away the complexities of distributed memory management.
*'''Scalability:''' Supports large-scale distributed applications.
*'''Consistency Models:''' Defines how and when changes in memory are visible to other nodes.
*'''Fault Tolerance:''' Can recover from failures by replicating memory across nodes.
==Memory Consistency Models==
Different consistency models define how updates to shared memory are observed:
{| class="wikitable"
!Consistency Model!!Description!!Example Use Case
|-
|'''Strict Consistency'''||Updates are immediately visible to all nodes.||Ideal but impractical in real-world distributed systems.
|-
|'''Sequential Consistency'''||All operations appear in some sequential order across all nodes.||Shared-memory multiprocessors.
|-
|'''Causal Consistency'''||Ensures that causally related memory updates appear in order.||Event-driven systems.
|-
|'''Release Consistency'''||Updates are visible after a synchronization operation (e.g., lock release).||High-performance computing.
|}
==Implementation Approaches==
Distributed shared memory can be implemented using different techniques:
*'''Hardware-Based DSM:''' Implements shared memory at the hardware level (e.g., NUMA systems).
*'''Software-Based DSM:''' Uses middleware or runtime systems to manage shared memory.
*'''Hybrid DSM:''' Combines hardware and software techniques for efficiency.
==Example DSM Systems==
*'''TreadMarks:''' A software DSM system supporting relaxed memory consistency.
*'''Memcached:''' A distributed caching system that can function as a shared memory abstraction.
*'''Spark RDDs:''' In-memory data sharing in distributed computing environments.
*'''IBM Coherence:''' A distributed in-memory data grid.
==Comparison with Other Memory Models==
{| class="wikitable"
!Feature!!Distributed Shared Memory!!Message Passing!!Centralized Shared Memory
|-
|'''Abstraction Level'''||High||Low||High
|-
|'''Communication Method'''||Implicit memory access||Explicit message exchange||Direct memory access
|-
|'''Performance'''||Moderate||High||High
|-
|'''Fault Tolerance'''||High (via replication)||High (via redundancy)||Low
|}
==Advantages==
*Simplifies parallel programming by abstracting memory distribution.
*Reduces the need for explicit communication mechanisms.
*Provides a scalable solution for distributed computing applications.
==Limitations==
*Memory consistency overhead can degrade performance.
*Higher latency compared to local shared memory due to network communication.
*Complexity in ensuring fault tolerance and replication.
==Applications==
*'''High-Performance Computing (HPC):''' Used in large-scale parallel computing systems.
*'''Distributed Databases:''' Enables shared access to distributed in-memory storage.
*'''Cloud Computing:''' Used in distributed caching and shared resource management.
*'''Big Data Processing:''' Facilitates in-memory data sharing in frameworks like Apache Spark.
==See Also==
*[[Shared Memory]]
*[[Message Passing Interface (MPI)]]
*[[Parallel Computing]]
*[[Cloud Computing]]
*[[Big Data Processing]]
*[[NUMA (Non-Uniform Memory Access)]]
[[분류:Distributed Computing]]

Piccolo

2025-02-01T01:52:45Z

Matei: 새 문서: '''Piccolo''' is a distributed in-memory computing framework designed to simplify the development of parallel applications. It provides a shared, distributed key-value store that allows workers to efficiently process large datasets while reducing communication overhead. ==Overview== Piccolo enables efficient distributed computing by: *'''In-Memory Data Storage:''' Uses a distributed key-value store to minimize disk I/O. *'''Fine-Grained Data Sharing:''' Allows workers to share s...

'''Piccolo''' is a distributed in-memory computing framework designed to simplify the development of parallel applications. It provides a shared, distributed key-value store that allows workers to efficiently process large datasets while reducing communication overhead.
==Overview==
Piccolo enables efficient distributed computing by:
*'''In-Memory Data Storage:''' Uses a distributed key-value store to minimize disk I/O.
*'''Fine-Grained Data Sharing:''' Allows workers to share state via a global table abstraction.
*'''Fault Tolerance:''' Supports recovery mechanisms to handle worker failures.
*'''Efficient Synchronization:''' Reduces communication overhead through user-defined consistency models.
Piccolo provides a programming model where developers can focus on computation while the framework handles data distribution and consistency.
==Key Features==
*'''Shared Global Tables''' – Workers access shared state stored in distributed key-value tables.
*'''Automatic Data Partitioning''' – Distributes data across workers for parallel processing.
*'''Flexible Consistency Models''' – Supports user-defined update models to balance performance and correctness.
*'''Fault Recovery''' – Can recover from worker failures by reloading lost state.
*'''Scalability''' – Designed to run efficiently on large clusters.
==How Piccolo Works==
#'''Workers Execute User Code:''' Each worker runs a computation task on distributed data.
#'''Global Tables Store State:''' Data is shared through distributed key-value tables.
#'''Synchronization Ensures Consistency:''' User-defined update models handle concurrent modifications.
#'''Checkpointing Provides Fault Tolerance:''' Periodic checkpoints allow recovery from failures.
==Example Usage==
A simple Piccolo job that counts word occurrences in a distributed manner:<syntaxhighlight lang="python">
import piccolo

# Define a distributed key-value table
word_counts = piccolo.Table("word_counts")

def count_words(worker, data):
for line in data:
for word in line.split():
word_counts.update(word, lambda x: x + 1 if x else 1)

# Run job on a distributed cluster
piccolo.run(count_words, input_data="hdfs://input.txt")
</syntaxhighlight>
==Comparison with Other Distributed Frameworks==
{| class="wikitable"
!Feature!!Piccolo!!Hadoop (MapReduce)!!Apache Spark
|-
|'''Data Storage'''||In-Memory Key-Value Store||Distributed File System||Resilient Distributed Datasets (RDDs)
|-
|'''Programming Model'''||Shared Global Tables||Map and Reduce Functions||Functional Transformations
|-
|'''Fault Tolerance'''||Checkpointing||Data Replication||Lineage-Based Recovery
|-
|'''Use Case'''||Iterative Data Processing||Batch Processing||Batch & Streaming Processing
|}
==Advantages==
*Faster than traditional MapReduce due to in-memory processing.
*Simple API for shared global state management.
*Scales efficiently for iterative computations.
==Limitations==
*Limited adoption compared to Spark and Hadoop.
*Not optimized for streaming workloads.
*Requires explicit consistency management by users.
==Applications==
*'''Graph Processing:''' Computing PageRank, social network analysis.
*'''Machine Learning:''' Distributed training of models with shared parameters.
*'''Iterative Computation:''' Workloads requiring frequent updates to shared state.
==See Also==
*[[Distributed Computing]]
*[[In-Memory Computing]]
*[[MapReduce]]
*[[Apache Spark]]
*[[Big Data Processing]]
[[분류:Distributed Computing]]

FlumeJava

2025-01-31T20:53:07Z

Matei: 새 문서: '''FlumeJava''' is a Java-based distributed data processing framework developed by Google for building and executing efficient, parallel, and distributed pipelines. It provides an abstraction over MapReduce and other parallel computation models, enabling users to write high-level data processing workflows. ==Overview== FlumeJava simplifies large-scale data processing by providing: *'''Lazy Evaluation:''' Pipelines are defined but not executed immediately, allowing for optimizati...

'''FlumeJava''' is a Java-based distributed data processing framework developed by Google for building and executing efficient, parallel, and distributed pipelines. It provides an abstraction over MapReduce and other parallel computation models, enabling users to write high-level data processing workflows.
==Overview==
FlumeJava simplifies large-scale data processing by providing:
*'''Lazy Evaluation:''' Pipelines are defined but not executed immediately, allowing for optimization before execution.
*'''Parallel Execution:''' Supports distributed computation over large datasets.
*'''Pipeline Abstraction:''' Enables users to write composable data transformations without handling low-level MapReduce details.
FlumeJava is designed to improve productivity by allowing developers to focus on pipeline logic rather than parallel execution mechanics.
==Key Features==
*'''High-Level API''' – Provides abstractions for common data transformations.
*'''Automatic Optimization''' – Lazily builds an execution plan and optimizes before running.
*'''Integration with MapReduce''' – Executes jobs on Google’s distributed infrastructure.
*'''Fault Tolerance''' – Handles failures efficiently during execution.
*'''Scalability''' – Processes petabyte-scale data efficiently.
==How FlumeJava Works==
#'''Define a Pipeline:''' The user writes a sequence of transformations using FlumeJava's API.
#'''Lazy Evaluation:''' The system constructs a deferred execution plan.
#'''Optimization:''' The execution plan is optimized before running.
#'''Execution:''' The optimized plan is executed on a distributed backend like MapReduce.
==Example Usage==
A simple FlumeJava pipeline for processing text data:<syntaxhighlight lang="java">
PCollection<String> lines = readTextFile("gs://input-data");
PCollection<String> words = lines.parallelDo(new DoFn<String, String>() {
public void process(String line, EmitFn<String> emitter) {
for (String word : line.split("\\s+")) {
emitter.emit(word);
}
}
}, stringType());

PCollection<KV<String, Integer>> wordCounts = words.count();
writeTextFile(wordCounts, "gs://output-data");
</syntaxhighlight>
==Comparison with Other Distributed Frameworks==
{| class="wikitable"
!Feature!!FlumeJava!!Apache Beam!!Hadoop (MapReduce)
|-
|'''Programming Model'''||High-Level Java API||Unified batch and streaming||Low-Level MapReduce API
|-
|'''Execution'''||Optimized pipeline execution||Portable across runners||Sequential execution
|-
|'''Ease of Use'''||High||High||Low
|-
|'''Primary Use Case'''||Batch Processing||Batch & Streaming||Batch Processing
|}
==Advantages==
*Provides a simple and expressive API for defining data pipelines.
*Automatically optimizes execution plans before running.
*Scales efficiently for large datasets.
==Limitations==
*Tightly integrated with Google’s ecosystem.
*Less flexible compared to newer frameworks like Apache Beam.
*No real-time streaming support (focused on batch processing).
==Applications==
*'''Log Processing:''' Analyzing large-scale system logs.
*'''ETL Pipelines:''' Extracting, transforming, and loading data.
*'''Machine Learning Data Preparation:''' Preprocessing large datasets for training models.
==See Also==
*[[Apache Beam]]
*[[MapReduce]]
*[[Distributed Computing]]
*[[Big Data Processing]]
*[[Google Cloud Dataflow]]
*[[ETL Process]]
[[분류:Distributed Computing]]

DryadLINQ

2025-01-31T20:44:44Z

Matei: 새 문서: '''DryadLINQ''' is a distributed computing framework developed by Microsoft that extends LINQ (Language Integrated Query) to work with large-scale data processing using the Dryad execution engine. It allows users to write data-parallel computations in C# or other .NET languages while leveraging distributed computing resources. ==Overview== DryadLINQ simplifies distributed data processing by combining: *'''Dryad''': A distributed execution engine that processes dataflow graphs ac...

'''DryadLINQ''' is a distributed computing framework developed by Microsoft that extends LINQ (Language Integrated Query) to work with large-scale data processing using the Dryad execution engine. It allows users to write data-parallel computations in C# or other .NET languages while leveraging distributed computing resources.
==Overview==
DryadLINQ simplifies distributed data processing by combining:
*'''Dryad''': A distributed execution engine that processes dataflow graphs across multiple machines.
*'''LINQ''': A high-level declarative programming model used for querying and manipulating data in .NET applications.
It enables developers to write parallel processing jobs in a familiar LINQ syntax, without requiring deep knowledge of distributed systems.
==Key Features==
*'''Seamless Integration with LINQ''' – Enables developers to write queries using LINQ while automatically distributing computations.
*'''Distributed Execution''' – Uses clusters of machines to execute data-parallel computations efficiently.
*'''Automatic Optimization''' – Translates LINQ queries into optimized execution graphs for parallel processing.
*'''Fault Tolerance''' – Supports recovery mechanisms in case of node failures.
*'''Scalability''' – Works efficiently with large datasets by distributing workloads dynamically.
==How DryadLINQ Works==
#'''User writes a LINQ query.'''
#*The developer writes a LINQ query using C# or another .NET language.
#'''DryadLINQ transforms the query.'''
#*The query is translated into a directed acyclic graph (DAG) representing the execution flow.
#'''Dryad executes the graph.'''
#*The Dryad engine schedules and executes the computation across a distributed cluster.
#'''Results are aggregated.'''
#*The final results are returned to the user after parallel execution completes.
==Example Usage==
A simple DryadLINQ query to process distributed data:<syntaxhighlight lang="csharp">
IQueryable<int> data = DistributedSource<int>.FromFile("input.txt");
var result = from num in data
where num % 2 == 0
select num * num;
result.ToDistributedStream("output.txt");
</syntaxhighlight>
==Comparison with Other Distributed Frameworks==
{| class="wikitable"
!Feature!!DryadLINQ!!Hadoop (MapReduce)!!Apache Spark
|-
|'''Programming Model'''||LINQ (Declarative)||Java/Python (Procedural)||RDDs, DataFrames (Functional)
|-
|'''Execution Model'''||Directed Acyclic Graph (DAG)||Map and Reduce Functions||DAG-based in-memory processing
|-
|'''Fault Tolerance'''||Checkpointing and recomputation||Data replication||Lineage-based recomputation
|-
|'''Ease of Use'''||High (familiar LINQ syntax)||Moderate (requires custom MapReduce logic)||High (functional programming model)
|}
==Advantages==
*Familiar syntax for .NET developers.
*Efficient distributed execution using Dryad’s DAG-based scheduler.
*Automatic query optimization and parallelization.
==Limitations==
*Limited adoption compared to Hadoop and Spark.
*Tightly integrated with the .NET ecosystem.
*Not actively maintained as Microsoft shifted focus to Azure-based big data solutions.
==Applications==
*'''Large-scale data analysis.'''
*'''Machine learning preprocessing.'''
*'''Log processing in distributed environments.'''
==See Also==
*[[LINQ]]
*[[Dryad (computing)]]
*[[Apache Spark]]
*[[Hadoop MapReduce]]
*[[Parallel Computing]]
*[[Big Data Processing]]
[[분류:Distributed Computing]]