Yrurtj

$ echo "zebra ant spider spider ant zebra ant"  | awk -v RS="[ n]+" '!n[$0]++' 

zebra

ant

spider

With tr, sort and uniq

echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq

or

echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq | xargs 

to get one line

With gnu sed:

sed ':s;s/(<S*>)(.*)<1>/12/g;ts'

You may add ;s/ */ /g to remove dublicate spaces.

Functions like this: If a word is a second time in this line, remove it and start over until no dublication is found anymore.

perl -lane '$,=$";print grep { ! $h{$_}++ } @F'

Obligatory awk solution:

$ echo "ant zebra ant spider spider ant zebra ant" | 

   awk -vRS=" " -vORS=" " '!a[$1] {a[$1]++} END{ for (x in a) print x;  } ' ; echo

zebra ant spider 

(The final echo is there for the newline)

Python

Option 1

#!/usr/bin/env python

# get_unique_words.py



import sys



l = 

for w in sys.argv[1].split(','):

  if w not in l:

    l += [ w ]

print ','.join(l)

Make executable, then call from Bash:

$ ./get_unique_words.py "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"

aaa,bbb,ccc

Or you could implement it as a Bash function, but the syntax is messy.

get_unique_words(){

  python -c "

l = 

for w in '$1'.split(','):

  if w not in l:

    l += [ w ]

print ','.join(l)"

}

Option 2

This option can become a one-liner if needed:

#!/usr/bin/env python

# get_unique_words.py



import sys



s_in = sys.argv[1]

l_in = s_in.split(',') # Turn string into a list.

set_out = set(l_in) # Turning a list into a set removes duplicates items.

s_out = ','.join(set_out) 

print s_out

In Bash:

get_unique_words(){

  python -c "print ','.join(set('$1'.split(',')))"

}

cat filename | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }' > newfile

Using the original tabular data in the file called file:

sed '1d' file | sort -u |

awk '{ color[$1] = ( color[$1] == "" ? $3 : color[$1] "," $3 ) }

     END { for (user in color) print user, color[user] }'

This generates

CCC red

BBB blue,red

AAA black,blue,green,red

The three steps of the pipeline:

1. The sed command removes the first line which is a header that we don't want to read.


2. 
   

   The sort command gives us unique lines. The sample data after the sort looks like

   
   
   

   AAA         | black
   
   AAA         | blue
   
   AAA         | green
   
   AAA         | red
   
   BBB         | blue
   
   BBB         | red
   
   CCC         | red
   
   

   
   


3. The awk command takes this data and produces a comma-delimited string for each user in the array color (where the username is the key into the array). At the end (in the END block), all collected data is outputted.

a="aaa aaa aaa bbb bbb ccc bbb ccc"

for item in $a

do

   echo $item

done | sort -u | (while read i; do ans="$ans $i"; done ; echo $ans)