Unable to connect to Hbase shell

Problem :

smartechies.mac $ hbase shell

ArgumentError: wrong number of arguments (0 for 1)
method_added at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10
method_added at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129
Pattern at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2
(root) at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1
require at org/jruby/RubyKernel.java:1062
(root) at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42
(root) at /usr/local/Cellar/hbase/1.2.6/libexec/bin/../bin/hirb.rb:38

Solution : 

Point the JAVA_HOME variable to correct value in  conf/hbase-env.sh

Example :

smartechies.mac $ cat ~/.profile | grep JAVA_HOME
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home
#export JAVA_HOME=/usr/libexec/java_home

smartechies.mac $ cat /usr/local/Cellar/hbase/1.2.6/libexec/conf/hbase-env.sh | grep JAVA_HOME
#export JAVA_HOME=”$(/usr/libexec/java_home)”


smartechies.mac$ hbase shell
2018-04-11 12:14:48,806 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.8.0/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter ‘help<RETURN>’ for list of supported commands.
Type “exit<RETURN>” to leave the HBase Shell
Version 1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017

hbase(main) :001 >


Create Superuser in HUE

[ec2-user@ip-123-45-67-890 ~]$ ls -ltr /usr/lib/hue/build/env/bin/hue

-rwxr-xr-x 1 root root 523 Sep 22 22:09 /usr/lib/hue/build/env/bin/hue

[ec2-user@ip-123-45-67-890 ~]$ sudo /usr/lib/hue/build/env/bin/hue createsuperuser

Username (leave blank to use ‘root’): sudhir

Email address: mail2sudhir.online@gmail.com


Password (again):

Superuser created successfully.

[ec2-user@ip-123-45-67-890 ~]$

Solved : error “/usr/bin/env: node: No such file or directory

Error :

“/usr/bin/env: node: No such file or directory

Solution :

ln -s /usr/bin/nodejs /usr/bin/node


Followup issue :

Error: EACCES: permission denied, open ‘/home/ubuntu/.config/configstore/bower-github.json’

Solved (click here for solution)

Solved: permission denied, open ‘/home/ubuntu/.config/configstore/bower-github.json’ in ubuntu

Error :

Error: EACCES: permission denied, open ‘/home/ubuntu/.config/configstore/bower-github.json’
You don’t have access to this file.

at Error (native)
at Object.fs.openSync (fs.js:549:18)
at Object.fs.readFileSync (fs.js:397:15)
at Object.create.all.get (/usr/local/lib/node_modules/bower/lib/node_modules/configstore/index.js:35:26)
at Object.Configstore (/usr/local/lib/node_modules/bower/lib/node_modules/configstore/index.js:28:44)
at readCachedConfig (/usr/local/lib/node_modules/bower/lib/config.js:19:23)
at defaultConfig (/usr/local/lib/node_modules/bower/lib/config.js:11:12)
at Object.<anonymous> (/usr/local/lib/node_modules/bower/lib/index.js:16:32)
at Module._compile (module.js:410:26)
at Object.Module._extensions..js (module.js:417:10)


Solution :

sudo chown -R $USER:$GROUP ~/.npm
sudo chown -R $USER:$GROUP ~/.config

Solved : Could not determine java version when running gradle

Error :

21:04:36.791 [INFO] [org.gradle.internal.nativeintegration.services.NativeServices] Initialized native services in: /home/ubuntu/.gradle/native
21:04:36.817 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Could not determine java version from ‘9.0.4’.
21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is:
21:04:36.823 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] java.lang.IllegalArgumentException: Could not determine java version from ‘9.0.4’.
21:04:36.823 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.api.JavaVersion.toVersion(JavaVersion.java:70)
21:04:36.823 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.api.JavaVersion.current(JavaVersion.java:80)
21:04:36.823 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.internal.jvm.UnsupportedJavaRuntimeException.assertUsingVersion(UnsupportedJavaRuntimeException.java:29)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:32)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:210)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:174)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.Main.doAction(Main.java:33)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
21:04:36.824 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
21:04:36.825 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at java.base/java.lang.reflect.Method.invoke(Method.java:564)
21:04:36.825 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:60)
21:04:36.825 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:37)
21:04:36.825 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.launcher.GradleMain.main(GradleMain.java:23)


Solution :

spradhan $ vi gradle/wrapper/gradle-wrapper.properties

distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists distributionUrl=https\://services.gradle.org/distributions/gradle-4.3-src.zip

Update the part marked as RED to -all or -bin

spradhan $ vi gradle/wrapper/gradle-wrapper.properties

distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists distributionUrl=https\://services.gradle.org/distributions/gradle-4.3all.zip

ERROR when writing file to S3 bucket from EMRFS enabled Spark cluster


18/03/02 01:42:17 INFO RetryInvocationHandler: Exception while invoking ConsistencyCheckerS3FileSystem.mkdirs over null. Retrying after sleeping for 10000ms. com.amazon.ws.emr.hadoop.fs.consistency.exception.ConsistencyException: Directory ‘bucket/folder/_temporary’ present in the metadata but not s3 at com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.getFileStatus(ConsistencyCheckerS3FileSystem.java:506)


Root cause :

Mostly the consistent problem comes due to

  • Manual deletion of files and directory from S3 console
  • retry logic in spark and hadoop systems.
  • When a process of creating a file on s3 failed, but it already updated in the dynamodb.
  • when the hadoop process restarts the process as the entry is already present in the dynamodb. It throws the consistent error.

Solution :

Try re-run your spark job by cleaning up the EMRFS metadata in dynamo db.

Follow the steps to clean-up & Restore the indended specific directory in the S3 bucket….


Deletes all the objects in the path, emrfs delete uses the hash function to delete the records, so it may delete unwanted entries also, so we are doing the import and sync in the consequent steps

Delete all the metadata

emrfs delete   s3://<bucket>/path

Retrieves the metadata for the objects that are physically present in s3 into dynamo db

emrfs import s3://<bucket>/path 

Sync the data between s3 and the metadata.

emrfs sync s3://<bucket>/path 

After all the operations, to see whether that particular object is present in both s3 and metadata

emrfs diff s3://<bucket>/path 

Amazon Aurora MySql Commands line

Connecting to a Database on a DB Instance Running the MySQL Database Engine

Once Amazon RDS provisions your DB instance, you can use any standard SQL client application to connect to a database on the DB instance. In this example, you connect to a database on a MySQL DB instance using MySQL monitor commands. One GUI-based application you can use to connect is MySQL Workbench; for more information, go to the Download MySQL Workbench page. For more information on using MySQL, go to the MySQL documentation.


To connect to a database on a DB instance using MySQL monitor

  1. Find the endpoint (DNS name) and port number for your DB Instance.
    1. Open the RDS console and then choose Instances to display a list of your DB instances.
    2. Choose the MySQL DB instance and choose See details from Instance actions to display the details for the DB instance.
    3. Scroll to the Connect section and copy the endpoint. Also, note the port number. You need both the endpoint and the port number to connect to the DB instance.Screen Shot 2018-02-27 at 11.46.17 AM
  2. Type the following command at a command prompt on a client computer to connect to a database on a MySQL DB instance using the MySQL monitor. Substitute the DNS name for your DB instance for <endpoint>, the master user name you used for <mymasteruser>, and the master password you used for <password>.
    spradhan.macosx$ mysql -h <endpoint> -P 3306 -u <mymasteruser>> -p

    You should see output similar to the following.

    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 350
    Server version: 5.6.27-log MySQL Community Server (GPL)
    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.


Basic MySQL Commandline operations

To login (from unix shell) use -h only if needed.
# [mysql dir]/bin/mysql -h hostname -u username -ppassword
To login (from windows)
mysql dir/bin/mysql.exe -h hostname -u username -ppassword

Create a database.
mysql> create database [databasename];

List all databases on the server.
mysql> show databases;

Switch to a database.
mysql> use [db name];

To see all the tables in the db.
mysql> show tables;

To see table’s field formats.
mysql> describe [table name];

To delete a db.
mysql> drop database [database name];

To delete a table.
mysql> drop table [table name];

Show all data from a table.
mysql> SELECT * FROM [table name];

To return columns and column information.
mysql> show columns from [table name];

Show particular rows with the given value.
mysql> SELECT * FROM [table name] WHERE [field name] = “value”;

Show all records containing the name “Something” AND the phone number ‘0123456789’.
mysql> SELECT * FROM [table name] WHERE name = “Something” AND phone_number = ‘0123456789’;

Show all records not containing the name “Something” AND the phone number ‘0123456789’ order by the phone_number field.
mysql> SELECT * FROM [table name] WHERE name != “Something” AND phone_number = ‘0123456789’ order by phone_number;

Show all records starting with the letters ‘Something‘ AND the phone number ‘0123456789’.
mysql> SELECT * FROM [table name] WHERE name like “Something%” AND phone_number = ‘0123456789’;

Show all records starting with letters ‘Something’ AND the phone number ‘0123456789’ limit to records 1 through 5.
mysql> SELECT * FROM [table name] WHERE name like “Something%” AND phone_number = ‘0123456789‘ limit 1,5;

Use a regular expression to find records. Use “REGEXP BINARY” to force case-sensitivity. This finds any record beginning with a.
mysql> SELECT * FROM [table name] WHERE rec RLIKE “^a”;

Show unique records.
mysql> SELECT DISTINCT [column name] FROM [table name];

Show selected records sorted in an ascending (asc) or descending (desc).
mysql> SELECT [col1],[col2] FROM [table name] ORDER BY [col2] DESC;

Return number of rows.
mysql> SELECT COUNT(*) FROM [table name];

Sum column.
mysql> SELECT SUM(*) FROM [table name];

Creating a new user. Login as root. Switch to the MySQL db. Make the user. Update privs.
# mysql -u root -p
mysql> use mysql;
mysql> INSERT INTO user (Host,User,Password) VALUES(‘%’,’username’,PASSWORD(‘password’));
mysql> flush privileges;

Change a users password from unix shell.
# [mysql dir]/bin/mysqladmin -u username -h hostname -ppassword ‘new-password’

Change a users password from MySQL prompt. Login as root. Set the password. Update privileges.
# mysql -u root -p
mysql> SET PASSWORD FOR ‘user’@’hostname’ = PASSWORD(‘password’);
mysql> flush privileges;

Recover a MySQL root password :

  1. Stop the MySQL server process.
  2. Start again with no grant tables.
  3. Login to MySQL as root. Set new password.
  4. Exit MySQL and restart MySQL server.
    # /etc/init.d/mysql stop
    # mysqld_safe –skip-grant-tables
    # mysql -u root
    mysql> use mysql;
    mysql> update user set password=PASSWORD(“newpassword”) where User=’root’;
    mysql> flush privileges;
    mysql> quit
    # /etc/init.d/mysql stop
    # /etc/init.d/mysql start

Set a root password if there is no root password.
# mysqladmin -u root password newpassword

Update a root password.
# mysqladmin -u root -p oldpassword newpassword

Allow the user “Someone” to connect to the server from localhost using the password “passwd”. Login as root. Switch to the MySQL db. Give privs. Update privs.
# mysql -u root -p
mysql> use mysql;
mysql> grant usage on *.* to Someone@localhost identified by ‘passwd’;
mysql> flush privileges;

Give user privilages for a db. Login as root. Switch to the MySQL db. Grant privs. Update privs.
# mysql -u root -p
mysql> use mysql;
mysql>INSERT INTO user(Host,Db,User,Select_priv,Insert_priv,Update_priv,Delete_priv,Create_priv,Drop_priv)
VALUES (‘%’,’databasename’,’username’,’Y’,’Y’,’Y’,’Y’,’Y’,’N’);
mysql> flush privileges;


mysql> grant all privileges on databasename.* to username@localhost;
mysql> flush privileges;

To update info already in a table.
mysql> UPDATE [table name] SET Select_priv = ‘Y’,Insert_priv = ‘Y’,Update_priv = ‘Y’ where [field name] = ‘user’;

Delete a row(s) from a table.
mysql> DELETE from [table name] where [field name] = ‘fieldvalue’;

Update database permissions/privilages.
mysql> flush privileges;

Delete a column.
mysql> alter table [table name] drop column [column name];

Add a new column to db.
mysql> alter table [table name] add column [new column name] varchar (20);

Change column name.
mysql> alter table [table name] change [old column name] [new column name] varchar (50);

Make a unique column so you get no dupes.
mysql> alter table [table name] add unique ([column name]);

Make a column bigger.
mysql> alter table [table name] modify [column name] VARCHAR(3);

Delete unique from table.
mysql> alter table [table name] drop index [colmn name];

Load a CSV file into a table.
mysql> LOAD DATA INFILE ‘/tmp/filename.csv’ replace INTO TABLE [table name] FIELDS TERMINATED BY ‘,’ LINES TERMINATED BY ‘\n’ (field1,field2,field3);

Dump all databases for backup. Backup file is sql commands to recreate all db’s.
# mysqldump -u username -ppassword –opt > /tmp/alldatabases.sql

Dump one database for backup.
# mysqldump -u username -ppassword –databases databasename > /tmp/databasename.sql

Dump a table from a database.
# mysqldump -u username -ppassword databasename tablename > /tmp/databasename.tablename.sql

Restore database (or database table) from backup.
# mysql -u username -ppassword databasename < /tmp/databasename.sql

Create Table Example 1.
mysql> CREATE TABLE employee (name VARCHAR(20));

Create Table Example 2.
mysql> create table person (personid int(50) not null auto_increment primary key,firstname varchar(35),middlename varchar(50),lastnamevarchar(50) default ‘UNKNOWN’);