Hi Eva.
I took a look at the 1880 census for Dudley Smith, and then noted his neighbors in order to find him in 1870, since he wasn't listed under that name. Were you aware that your ancestors were WILLIAMS?
Next door to Dudley in 1880 was the family of SQUIRE WILLIAMS and in checking them in 1870, I found something interesting. Here are his census:

1880 Duplin County, North Carolina, Smiths Twp,Dist.84,Image 4, #31/31:
SQUIRE WILLIAMS - B - 50 - M - FARMING - NC NC NC
SUSAN WILLIAMS (WIFE) - B - 54 - M - KEEPING HOUSE -NC NC NC
LUCY J.WILLIAMS (DAU) - B - 36 - S - AT HOME - NC NC NC
GEORGE A.WILLIAMS(SON)- B - 21 - S - FARM LABORER - NC NC NC
ABRAM WILLIAMS (SON) - B - 20 - S - WORKS ON FARM - NC NC NC
SQUIRE WILLIAMS JR.(SON)-B- 18 - S - WORKS ON FARM - NC NC NC

1870 Duplin County,North Carolina,Smith Twp,Image 7, #47/47:
SQUIRE WILLIAMS - 55 - B - FARM LABORER - NC
SUSAN WILLIAMS - 50 - B - KEEPING HOUSE- NC
ALICE WILLIAMS - 23 - B - FARM LABORER - NC
JANE WILLIAMS - 21 - B - FARM LABORER - NC
DUNDEE WILLIAMS - 19 - B - FARM LABORER - NC (male)
PRISCILLA WILLIAMS-17- B - FARM LABORER - NC
CASSA A. WILLIAMS -15- B - AT HOME - NC (female)
GEORGIANA WILLIAMS -13 - B- AT HOME - NC
FRANCIS WILLIAMS - 11 - B - AT HOME - NC (female)
ABRAHAM WILLIAMS - 9 - B - AT HOME - NC
SQUIRE WILLIAMS - 5 - B - AT HOME - NC

(I'd say that LUCY J. and JANE were the same person, and that DUNDEE is probably the same person as DUDLEY. PRISCILLA could either have been his sister or wife, listed out of order. As you must know, DUDLEY & PRISCILLA were aged 30 & 28,respectively, in 1880)

There was also another family next door in 1870 with a PRISCILLA who could have possibly later been DUDLEY's wife.
#48/48:
ABSALOM OUTLAW - 53 - B - FARM LABORER - 100 - NC
CHARITY OUTLAW - 48 - B - KEEPING HOUSE - NC
MAYO? OUTLAW - 21 - B - FARM LABORER - NC (male)
FRANK OUTLAW - 18 - B - FARM LABORER - NC
PRISCILLA OUTLAW - 16 - B - FARM LABORER - NC
CAROLINE OUTLAW - 14 - B - FARM LABORER - NC
HANNAH OUTLAW - 12 - B - AT HOME - NC
ANN OUTLAW - 10 - B - AT HOME - NC
PERDIDO OUTLAW - 8 - B - AT HOME - NC (female)
EVERETT OUTLAW - 1 - B - AT HOME - NC
HOLLAND OUTLAW - 80 -B - AT HOME - NC (female)

A few doors away in 1870 you have several white families which are good prospects for being the former slaveholders:
IVY SMITH,male aged 50
ZACHIUS SMITH, male, age 58
EDWARD SMITH, aged 55
BRANCH WILLIAMS, male, aged 36

Those are the families that you need to start looking for wills, deeds, estate records, inventories, etc. in order to start tracing your family back even further.

Also, here are some Marriages for Duplin County that you might be interested in from "Somebody Knows My Name, Marriages of Freed People in North Carolina", by Barnetta McGhee White:

(the length of time married is approximately as of 1866. Ms. White also indicates that she left the females surname blank when it was the same as the males)
SQUIRE WILLIAMS/ADLINE PEARSALL - Married for 3 years
SQUIRE WILLIAMS/PRISCILLA ------- - Married for 40 years
SQUIRE WILLIAMS/SUSAN ---------- - Married 21 years

ABSALOM SMITH/CHARITY ------- - Married for 20 years

So, you see there was some name changing going on here, and you need to do further research on these families using all three of the surnames. You'll find it to be very useful to keep an eye on the neighbors when doing this research because so many times you'll find family members intermarried with them.
I hope this helps you some and gets you started in some new directions for locating more about your ancestors.

Deloris