You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
64 lines
2.1 KiB
64 lines
2.1 KiB
5 years ago
|
[](https://gitter.im/big-data-europe/Lobby)
|
||
|
|
|
||
|
|
# Changes
|
||
|
|
|
||
|
|
Version 2.0.0 introduces uses wait_for_it script for the cluster startup
|
||
|
|
|
||
|
|
# Hadoop Docker
|
||
|
|
|
||
|
|
## Supported Hadoop Versions
|
||
|
|
See repository branches for supported hadoop versions
|
||
|
|
|
||
|
|
## Quick Start
|
||
|
|
|
||
|
|
To deploy an example HDFS cluster, run:
|
||
|
|
```
|
||
|
|
docker-compose up
|
||
|
|
```
|
||
|
|
|
||
|
|
Run example wordcount job:
|
||
|
|
```
|
||
|
|
make wordcount
|
||
|
|
```
|
||
|
|
|
||
|
|
Or deploy in swarm:
|
||
|
|
```
|
||
|
|
docker stack deploy -c docker-compose-v3.yml hadoop
|
||
|
|
```
|
||
|
|
|
||
|
|
`docker-compose` creates a docker network that can be found by running `docker network list`, e.g. `dockerhadoop_default`.
|
||
|
|
|
||
|
|
Run `docker network inspect` on the network (e.g. `dockerhadoop_default`) to find the IP the hadoop interfaces are published on. Access these interfaces with the following URLs:
|
||
|
|
|
||
|
|
* Namenode: http://<dockerhadoop_IP_address>:9870/dfshealth.html#tab-overview
|
||
|
|
* History server: http://<dockerhadoop_IP_address>:8188/applicationhistory
|
||
|
|
* Datanode: http://<dockerhadoop_IP_address>:9864/
|
||
|
|
* Nodemanager: http://<dockerhadoop_IP_address>:8042/node
|
||
|
|
* Resource manager: http://<dockerhadoop_IP_address>:8088/
|
||
|
|
|
||
|
|
## Configure Environment Variables
|
||
|
|
|
||
|
|
The configuration parameters can be specified in the hadoop.env file or as environmental variables for specific services (e.g. namenode, datanode etc.):
|
||
|
|
```
|
||
|
|
CORE_CONF_fs_defaultFS=hdfs://namenode:8020
|
||
|
|
```
|
||
|
|
|
||
|
|
CORE_CONF corresponds to core-site.xml. fs_defaultFS=hdfs://namenode:8020 will be transformed into:
|
||
|
|
```
|
||
|
|
<property><name>fs.defaultFS</name><value>hdfs://namenode:8020</value></property>
|
||
|
|
```
|
||
|
|
To define dash inside a configuration parameter, use triple underscore, such as YARN_CONF_yarn_log___aggregation___enable=true (yarn-site.xml):
|
||
|
|
```
|
||
|
|
<property><name>yarn.log-aggregation-enable</name><value>true</value></property>
|
||
|
|
```
|
||
|
|
|
||
|
|
The available configurations are:
|
||
|
|
* /etc/hadoop/core-site.xml CORE_CONF
|
||
|
|
* /etc/hadoop/hdfs-site.xml HDFS_CONF
|
||
|
|
* /etc/hadoop/yarn-site.xml YARN_CONF
|
||
|
|
* /etc/hadoop/httpfs-site.xml HTTPFS_CONF
|
||
|
|
* /etc/hadoop/kms-site.xml KMS_CONF
|
||
|
|
* /etc/hadoop/mapred-site.xml MAPRED_CONF
|
||
|
|
|
||
|
|
If you need to extend some other configuration file, refer to base/entrypoint.sh bash script.
|